Bdpcm-based image coding method and device therefor

ABSTRACT

An image decoding method according to the present document comprises the steps of: deriving a quantized transform coefficient for a current block on the basis of BDPCM; deriving a transform coefficient by performing dequantization on the quantized transform coefficient; and deriving a residual sample on the basis of the transform coefficient, wherein, when the BDPCM is applied to the current block, inverse non-separable transform is not applied to the transform coefficient.

TECHNICAL FIELD

The present disclosure relates generally to an image coding technologyand, more particularly, to an image coding method based on a BDPCM(blockdifferential pulse coded modulation) in an image coding system and anapparatus therefor.

RELATED ART

Nowadays, the demand for high-resolution and high-quality images/videossuch as 4K, 8K or more ultra high definition (UHD) images/videos hasbeen increasing in various fields. As the image/video data becomeshigher resolution and higher quality, the transmitted information amountor bit amount increases as compared to the conventional image data.Therefore, when image data is transmitted using a medium such as aconventional wired/wireless broadband line or image/video data is storedusing an existing storage medium, the transmission cost and the storagecost thereof are increased.

Further, nowadays, the interest and demand for immersive media such asvirtual reality (VR), artificial reality (AR) content or hologram, orthe like is increasing, and broadcasting for images/videos having imagefeatures different from those of real images, such as a game image isincreasing.

Accordingly, there is a need for a highly efficient image/videocompression technique for effectively compressing and transmitting orstoring, and reproducing information of high resolution and high qualityimages/videos having various features as described above.

SUMMARY

A technical aspect of the present disclosure is to provide a method andan apparatus for increasing image coding efficiency.

Another technical aspect of the present disclosure is to provide amethod and an apparatus for increasing efficiency in transform indexcoding in image coding based on a BDPCM.

Another technical aspect of the present disclosure is to provide amethod and an apparatus for increasing efficiency in transform skip flagcoding in image coding based on a BDPCM

Still another technical aspect of the present disclosure is to provide amethod and an apparatus for performing BDPCM coding for each lumacomponent or chroma component are provided.

According to an embodiment of the present disclosure, there is providedan image decoding method performed by a decoding apparatus. The methodmay include deriving quantized transform coefficients for a currentblock based on a BDPCM; deriving transform coefficients by performing adequantization on the quantized transform coefficients; and derivingresidual samples based on the transform coefficients; wherein when theBDPCM is applied to the current block, an inverse non-separabletransform is not applied to the transform coefficients.

When the BDPCM is applied to the current block, the value of thetransform index for the inverse non-separable transform that is appliedto the current block is inferred to be 0.

When the BDPCM is applied to the current block, the value of a transformskip flag indicating whether a transform is skipped in the current blockis inferred to be 0.

The BDPCM is individually applied to a luma block of the current blockor a chroma block of the current block, wherein when the BDPCM isapplied to the luma block, the transform index for the luma block is notreceived, and wherein when the BDPCM is applied to the chroma block, thetransform index for the chroma block is not received.

When the width of the current block is less than or equal to a firstthreshold and the height of the current block is less than or equal to asecond threshold, the BDPCM is applied to the current block.

Quantized transform coefficients are derived based on directioninformation on the direction in which the BDPCM is performed.

The image decoding method further comprises performing an intraprediction on the current block based on the direction in which theBDPCM is performed.

The direction information indicates a horizontal direction or a verticaldirection

According to another embodiment of the present disclosure, there isprovided an image encoding method performed by an encoding apparatus.The method may include: deriving prediction samples for a current blockbased on a BDPCM; deriving residual samples for the current block basedon the prediction samples; performing quantization on the residualsamples; deriving quantized residual information based on the BDPCM; andencoding the quantized residual information and coding information forthe current block; wherein when the BDPCM is applied to the currentblock, a non-separable transform is not applied to the current block.

According to still another embodiment of the present disclosure, theremay be provided a digital storage medium that stores image dataincluding encoded image information and a bitstream generated accordingto an image encoding method performed by an encoding apparatus.

According to yet another embodiment of the present disclosure, there maybe provided a digital storage medium that stores image data includingencoded image information and a bitstream to cause a decoding apparatusto perform the image decoding method.

According to the present disclosure, it is possible to increase overallimage/video compression efficiency.

According to the present disclosure, it is possible to increase overallimage/video compression efficiency in transform index coding.

According to the present disclosure, it is possible to increaseefficiency in transform index coding in image coding based on a BDPCM.

According to the present disclosure, it is possible to increaseefficiency in transform skip flag coding in image coding based on aBDPCM.

According to the present disclosure, a method and an apparatus forperforming BDPCM coding for each luma component or chroma component areprovided.

The effects that can be obtained through specific examples of thepresent disclosure are not limited to the effects listed above. Forexample, there may be various technical effects that a person havingordinary skill in the related art can understand or derive from thepresent disclosure. Accordingly, specific effects of the presentdisclosure are not limited to those explicitly described in the presentdisclosure and may include various effects that can be understood orderived from the technical features of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates an example of a video/image codingsystem to which the present disclosure is applicable.

FIG. 2 is a diagram schematically illustrating a configuration of avideo/image encoding apparatus to which the present disclosure isapplicable.

FIG. 3 is a diagram schematically illustrating a configuration of avideo/image decoding apparatus to which the present disclosure isapplicable.

FIG. 4 schematically illustrates a multiple transform techniqueaccording to an embodiment of the present disclosure.

FIG. 5 illustrates directional intra modes of 65 prediction directions.

FIG. 6 is a diagram illustrating an RST according to an embodiment ofthe present disclosure.

FIG. 7 is a flowchart illustrating an operation of an image decodingapparatus according to an embodiment of the present disclosure.

FIG. 8 is a control flowchart illustrating an image decoding methodaccording to an embodiment of the present disclosure.

FIG. 9 is a flowchart illustrating an operation of an image encodingapparatus according to an embodiment of the present disclosure.

FIG. 10 is a control flowchart illustrating an image encoding methodaccording to an embodiment of the present disclosure.

FIG. 11 illustrates the structure of a content streaming system to whichthe present disclosure is applied.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

While the present disclosure may be susceptible to various modificationsand include various embodiments, specific embodiments thereof have beenshown in the drawings by way of example and will now be described indetail. However, this is not intended to limit the present disclosure tothe specific embodiments disclosed herein. The terminology used hereinis for the purpose of describing specific embodiments only, and is notintended to limit technical idea of the present disclosure. The singularforms may include the plural forms unless the context clearly indicatesotherwise. The terms such as “include” and “have” are intended toindicate that features, numbers, steps, operations, elements,components, or combinations thereof used in the following descriptionexist, and thus should not be understood as that the possibility ofexistence or addition of one or more different features, numbers, steps,operations, elements, components, or combinations thereof is excluded inadvance.

Meanwhile, each component on the drawings described herein isillustrated independently for convenience of description as tocharacteristic functions different from each other, and however, it isnot meant that each component is realized by a separate hardware orsoftware. For example, any two or more of these components may becombined to form a single component, and any single component may bedivided into plural components. The embodiments in which components arecombined and/or divided will belong to the scope of the patent right ofthe present disclosure as long as they do not depart from the essence ofthe present disclosure.

Hereinafter, preferred embodiments of the present disclosure will beexplained in more detail while referring to the attached drawings. Inaddition, the same reference signs are used for the same components onthe drawings, and repeated descriptions for the same components will beomitted.

This document relates to video/image coding. For example, themethod/example disclosed in this document may relate to a VVC (VersatileVideo Coding) standard (ITU-T Rec. H.266), a next-generation video/imagecoding standard after VVC, or other video coding related standards(e.g., HEVC (High Efficiency Video Coding) standard (ITU-T Rec. H.265),EVC (essential video coding) standard, AVS2 standard, etc.).

In this document, a variety of embodiments relating to video/imagecoding may be provided, and, unless specified to the contrary, theembodiments may be combined to each other and be performed.

In this document, a video may mean a set of a series of images overtime. Generally a picture means a unit representing an image at aspecific time zone, and a slice/tile is a unit constituting a part ofthe picture. The slice/tile may include one or more coding tree units(CTUs). One picture may be constituted by one or more slices/tiles. Onepicture may be constituted by one or more tile groups. One tile groupmay include one or more tiles.

A pixel or a pel may mean a smallest unit constituting one picture (orimage). Also, ‘sample’ may be used as a term corresponding to a pixel. Asample may generally represent a pixel or a value of a pixel, and mayrepresent only a pixel/pixel value of a luma component or only apixel/pixel value of a chroma component. Alternatively, the sample mayrefer to a pixel value in the spatial domain, or when this pixel valueis converted to the frequency domain, it may refer to a transformcoefficient in the frequency domain.

A unit may represent the basic unit of image processing. The unit mayinclude at least one of a specific region and information related to theregion. One unit may include one luma block and two chroma (e.g., cb,cr) blocks. The unit and a term such as a block, an area, or the likemay be used in place of each other according to circumstances. In ageneral case, an M×N block may include a set (or an array) of samples(or sample arrays) or transform coefficients consisting of M columns andN rows.

In this document, the term “I” and “,” should be interpreted to indicate“and/or.” For instance, the expression “A/B” may mean “A and/or B.”Further, “A, B” may mean “A and/or B.” Further, “A/B/C” may mean “atleast one of A, B, and/or C.” Also, “A/B/C” may mean “at least one of A,B, and/or C.”

Further, in the document, the term “or” should be interpreted toindicate “and/or.” For instance, the expression “A or B” may include 1)only A, 2) only B, and/or 3) both A and B. In other words, the term “or”in this document should be interpreted to indicate “additionally oralternatively.”

In the present disclosure, “at least one of A and B” may mean “only A”,“only B”, or “both A and B”. In addition, in the present disclosure, theexpression “at least one of A or B” or “at least one of A and/or B” maybe interpreted as “at least one of A and B”.

In addition, in the present disclosure, “at least one of A, B, and C”may mean “only A”, “only B”, “only C”, or “any combination of A, B, andC”. In addition, “at least one of A, B, or C” or “at least one of A, B,and/or C” may mean “at least one of A, B, and C”.

In addition, a parenthesis used in the present disclosure may mean “forexample”. Specifically, when indicated as “prediction (intraprediction)”, it may mean that “intra prediction” is proposed as anexample of “prediction”. In other words, the “prediction” of the presentdisclosure is not limited to “intra prediction”, and “intra prediction”may be proposed as an example of “prediction”. In addition, whenindicated as “prediction (i.e., intra prediction)”, it may also meanthat “intra prediction” is proposed as an example of “prediction”.

Technical features individually described in one figure in the presentdisclosure may be individually implemented or may be simultaneouslyimplemented.

FIG. 1 schematically illustrates an example of a video/image codingsystem to which the present disclosure is applicable.

Referring to FIG. 1 , the video/image coding system may include a firstdevice (source device) and a second device (receive device). The sourcedevice may deliver encoded video/image information or data in the formof a file or streaming to the receive device via a digital storagemedium or network.

The source device may include a video source, an encoding apparatus, anda transmitter. The receive device may include a receiver, a decodingapparatus, and a renderer. The encoding apparatus may be called avideo/image encoding apparatus, and the decoding apparatus may be calleda video/image decoding apparatus. The transmitter may be included in theencoding apparatus. The receiver may be included in the decodingapparatus. The renderer may include a display, and the display may beconfigured as a separate device or an external component.

The video source may obtain a video/image through a process ofcapturing, synthesizing, or generating a video/image. The video sourcemay include a video/image capture device and/or a video/image generatingdevice. The video/image capture device may include, for example, one ormore cameras, video/image archives including previously capturedvideo/images, or the like. The video/image generating device mayinclude, for example, a computer, a tablet and a smartphone, and may(electronically) generate a video/image. For example, a virtualvideo/image may be generated through a computer or the like. In thiscase, the video/image capturing process may be replaced by a process ofgenerating related data.

The encoding apparatus may encode an input video/image. The encodingapparatus may perform a series of procedures such as prediction,transform, and quantization for compression and coding efficiency. Theencoded data (encoded video/image information) may be output in the formof a bitstream.

The transmitter may transmit the encoded video/image information or dataoutput in the form of a bitstream to the receiver of the receive devicethrough a digital storage medium or a network in the form of a file orstreaming. The digital storage medium may include various storagemediums such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, and the like. Thetransmitter may include an element for generating a media file through apredetermined file format, and may include an element for transmissionthrough a broadcast/communication network. The receiver mayreceive/extract the bitstream and transmit the received/extractedbitstream to the decoding apparatus.

The decoding apparatus may decode a video/image by performing a seriesof procedures such as dequantization, inverse transform, prediction, andthe like corresponding to the operation of the encoding apparatus.

The renderer may render the decoded video/image. The renderedvideo/image may be displayed through the display.

FIG. 2 is a diagram schematically illustrating a configuration of avideo/image encoding apparatus to which the present disclosure isapplicable. Hereinafter, what is referred to as the video encodingapparatus may include an image encoding apparatus.

Referring to FIG. 2 , the encoding apparatus 200 may include an imagepartitioner 210, a predictor 220, a residual processor 230, an entropyencoder 240, an adder 250, a filter 260, and a memory 270. The predictor220 may include an inter predictor 221 and an intra predictor 222. Theresidual processor 230 may include a transformer 232, a quantizer 233, adequantizer 234, an inverse transformer 235. The residual processor 230may further include a subtractor 231. The adder 250 may be called areconstructor or reconstructed block generator. The image partitioner210, the predictor 220, the residual processor 230, the entropy encoder240, the adder 250, and the filter 260, which have been described above,may be constituted by one or more hardware components (e.g., encoderchipsets or processors) according to an embodiment. Further, the memory270 may include a decoded picture buffer (DPB), and may be constitutedby a digital storage medium. The hardware component may further includethe memory 270 as an internal/external component.

The image partitioner 210 may partition an input image (or a picture ora frame) input to the encoding apparatus 200 into one or more processingunits. As one example, the processing unit may be called a coding unit(CU). In this case, starting with a coding tree unit (CTU) or thelargest coding unit (LCU), the coding unit may be recursivelypartitioned according to the Quad-tree binary-tree ternary-tree (QTBTTT)structure. For example, one coding unit may be divided into a pluralityof coding units of a deeper depth based on the quad-tree structure, thebinary-tree structure, and/or the ternary structure. In this case, forexample, the quad-tree structure may be applied first and thebinary-tree structure and/or the ternary structure may be applied later.Alternatively, the binary-tree structure may be applied first. Thecoding procedure according to the present disclosure may be performedbased on the final coding unit which is not further partitioned. In thiscase, the maximum coding unit may be used directly as a final codingunit based on coding efficiency according to the image characteristic.Alternatively, the coding unit may be recursively partitioned intocoding units of a further deeper depth as needed, so that the codingunit of an optimal size may be used as a final coding unit. Here, thecoding procedure may include procedures such as prediction, transform,and reconstruction, which will be described later. As another example,the processing unit may further include a prediction unit (PU) or atransform unit (TU). In this case, the prediction unit and the transformunit may be split or partitioned from the above-described final codingunit. The prediction unit may be a unit of sample prediction, and thetransform unit may be a unit for deriving a transform coefficient and/ora unit for deriving a residual signal from a transform coefficient.

The unit and a term such as a block, an area, or the like may be used inplace of each other according to circumstances. In a general case, anM×N block may represent a set of samples or transform coefficientsconsisting of M columns and N rows. The sample may generally represent apixel or a value of a pixel, and may represent only a pixel/pixel valueof a luma component, or only a pixel/pixel value of a chroma component.The sample may be used as a term corresponding to a pixel or a pel ofone picture (or image).

The subtractor 231 subtracts a prediction signal (predicted block,prediction sample array) output from the inter predictor 221 or theintra predictor 222 from an input image signal (original block, originalsample array) to generate a residual signal (residual block, residualsample array), and the generated residual signal is transmitted to thetransformer 232. In this case, as shown, a unit which subtracts theprediction signal (predicted block, prediction sample array) from theinput image signal (original block, original sample array) in theencoder 200 may be called the subtractor 231. The predictor may performprediction on a processing target block (hereinafter, referred to as‘current block’), and may generate a predicted block includingprediction samples for the current block. The predictor may determinewhether intra prediction or inter prediction is applied on a currentblock or CU basis. As discussed later in the description of eachprediction mode, the predictor may generate various information relatingto prediction, such as prediction mode information, and transmit thegenerated information to the entropy encoder 240. The information on theprediction may be encoded in the entropy encoder 240 and output in theform of a bitstream.

The intra predictor 222 may predict the current block by referring tosamples in the current picture. The referred samples may be located inthe neighbor of or apart from the current block according to theprediction mode. In the intra prediction, prediction modes may include aplurality of non-directional modes and a plurality of directional modes.The non-directional modes may include, for example, a DC mode and aplanar mode. The directional mode may include, for example, 33directional prediction modes or 65 directional prediction modesaccording to the degree of detail of the prediction direction. However,this is merely an example, and more or less directional prediction modesmay be used depending on a setting. The intra predictor 222 maydetermine the prediction mode applied to the current block by using theprediction mode applied to the neighboring block.

The inter predictor 221 may derive a predicted block for the currentblock based on a reference block (reference sample array) specified by amotion vector on a reference picture. At this time, in order to reducethe amount of motion information transmitted in the inter predictionmode, the motion information may be predicted on a block, subblock, orsample basis based on correlation of motion information between theneighboring block and the current block. The motion information mayinclude a motion vector and a reference picture index. The motioninformation may further include inter prediction direction (L0prediction, L1 prediction, Bi prediction, etc.) information. In the caseof inter prediction, the neighboring block may include a spatialneighboring block existing in the current picture and a temporalneighboring block existing in the reference picture. The referencepicture including the reference block and the reference pictureincluding the temporal neighboring block may be same to each other ordifferent from each other. The temporal neighboring block may be calleda collocated reference block, a collocated CU (colCU), and the like, andthe reference picture including the temporal neighboring block may becalled a collocated picture (colPic). For example, the inter predictor221 may configure a motion information candidate list based onneighboring blocks and generate information indicating which candidateis used to derive a motion vector and/or a reference picture index ofthe current block. Inter prediction may be performed based on variousprediction modes. For example, in the case of a skip mode and a mergemode, the inter predictor 221 may use motion information of theneighboring block as motion information of the current block. In theskip mode, unlike the merge mode, the residual signal may not betransmitted. In the case of the motion information prediction (motionvector prediction, MVP) mode, the motion vector of the neighboring blockmay be used as a motion vector predictor and the motion vector of thecurrent block may be indicated by signaling a motion vector difference.

The predictor 220 may generate a prediction signal based on variousprediction methods. For example, the predictor may apply intraprediction or inter prediction for prediction on one block, and, aswell, may apply intra prediction and inter prediction at the same time.This may be called combined inter and intra prediction (CIIP). Further,the predictor may be based on an intra block copy (IBC) prediction mode,or a palette mode in order to perform prediction on a block. The IBCprediction mode or palette mode may be used for content image/videocoding of a game or the like, such as screen content coding (SCC).Although the IBC basically performs prediction in a current block, itcan be performed similarly to inter prediction in that it derives areference block in a current block. That is, the IBC may use at leastone of inter prediction techniques described in the present disclosure.

The prediction signal generated through the inter predictor 221 and/orthe intra predictor 222 may be used to generate a reconstructed signalor to generate a residual signal. The transformer 232 may generatetransform coefficients by applying a transform technique to the residualsignal. For example, the transform technique may include at least one ofa discrete cosine transform (DCT), a discrete sine transform (DST), aKarhunen-Loeve transform (KLT), a graph-based transform (GBT), or aconditionally non-linear transform (CNT). Here, the GBT means transformobtained from a graph when relationship information between pixels isrepresented by the graph. The CNT refers to transform obtained based ona prediction signal generated using all previously reconstructed pixels.In addition, the transform process may be applied to square pixel blockshaving the same size or may be applied to blocks having a variable sizerather than the square one.

The quantizer 233 may quantize the transform coefficients and transmitthem to the entropy encoder 240, and the entropy encoder 240 may encodethe quantized signal (information on the quantized transformcoefficients) and output the encoded signal in a bitstream. Theinformation on the quantized transform coefficients may be referred toas residual information. The quantizer 233 may rearrange block typequantized transform coefficients into a one-dimensional vector formbased on a coefficient scan order, and generate information on thequantized transform coefficients based on the quantized transformcoefficients of the one-dimensional vector form. The entropy encoder 240may perform various encoding methods such as, for example, exponentialGolomb, context-adaptive variable length coding (CAVLC),context-adaptive binary arithmetic coding (CABAC), and the like. Theentropy encoder 240 may encode information necessary for video/imagereconstruction other than quantized transform coefficients (e.g. valuesof syntax elements, etc.) together or separately. Encoded information(e.g., encoded video/image information) may be transmitted or stored ona unit basis of a network abstraction layer (NAL) in the form of abitstream. The video/image information may further include informationon various parameter sets such as an adaptation parameter set (APS), apicture parameter set (PPS), a sequence parameter set (SPS), a videoparameter set (VPS) or the like. Further, the video/image informationmay further include general constraint information. In the presentdisclosure, information and/or syntax elements which aretransmitted/signaled to the decoding apparatus from the encodingapparatus may be included in video/image information. The video/imageinformation may be encoded through the above-described encodingprocedure and included in the bitstream. The bitstream may betransmitted through a network, or stored in a digital storage medium.Here, the network may include a broadcast network, a communicationnetwork and/or the like, and the digital storage medium may includevarious storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, andthe like. A transmitter (not shown) which transmits a signal output fromthe entropy encoder 240 and/or a storage (not shown) which stores it maybe configured as an internal/external element of the encoding apparatus200, or the transmitter may be included in the entropy encoder 240.

Quantized transform coefficients output from the quantizer 233 may beused to generate a prediction signal. For example, by applyingdequantization and inverse transform to quantized transform coefficientsthrough the dequantizer 234 and the inverse transformer 235, theresidual signal (residual block or residual samples) may bereconstructed. The adder 155 adds the reconstructed residual signal to aprediction signal output from the inter predictor 221 or the intrapredictor 222, so that a reconstructed signal (reconstructed picture,reconstructed block, reconstructed sample array) may be generated. Whenthere is no residual for a processing target block as in a case wherethe skip mode is applied, the predicted block may be used as areconstructed block. The adder 250 may be called a reconstructor or areconstructed block generator. The generated reconstructed signal may beused for intra prediction of a next processing target block in thecurrent block, and as described later, may be used for inter predictionof a next picture through filtering.

Meanwhile, in the picture encoding and/or reconstructing process, lumamapping with chroma scaling (LMCS) may be applied.

The filter 260 may improve subjective/objective video quality byapplying the filtering to the reconstructed signal. For example, thefilter 260 may generate a modified reconstructed picture by applyingvarious filtering methods to the reconstructed picture, and may storethe modified reconstructed picture in the memory 270, specifically inthe DPB of the memory 270. The various filtering methods may include,for example, deblocking filtering, sample adaptive offset, an adaptiveloop filter, a bilateral filter or the like. As discussed later in thedescription of each filtering method, the filter 260 may generatevarious information relating to filtering, and transmit the generatedinformation to the entropy encoder 240. The information on the filteringmay be encoded in the entropy encoder 240 and output in the form of abitstream.

The modified reconstructed picture which has been transmitted to thememory 270 may be used as a reference picture in the inter predictor221. Through this, the encoding apparatus can avoid prediction mismatchin the encoding apparatus 100 and a decoding apparatus when the interprediction is applied, and can also improve coding efficiency.

The memory 270 DPB may store the modified reconstructed picture in orderto use it as a reference picture in the inter predictor 221. The memory270 may store motion information of a block in the current picture, fromwhich motion information has been derived (or encoded) and/or motioninformation of blocks in an already reconstructed picture. The storedmotion information may be transmitted to the inter predictor 221 to beutilized as motion information of a neighboring block or motioninformation of a temporal neighboring block. The memory 270 may storereconstructed samples of reconstructed blocks in the current picture,and transmit them to the intra predictor 222.

FIG. 3 is a diagram schematically illustrating a configuration of avideo/image decoding apparatus to which the present disclosure isapplicable.

Referring to FIG. 3 , the image decoding apparatus 300 may include anentropy decoder 310, a residual processor 320, a predictor 330, an adder340, a filter 350 and a memory 360. The predictor 330 may include aninter predictor 331 and an intra predictor 332. The residual processor320 may include a dequantizer 321 and an inverse transformer 321. Theentropy decoder 310, the residual processor 320, the predictor 330, theadder 340, and the filter 350, which have been described above, may beconstituted by one or more hardware components (e.g., decoder chipsetsor processors) according to an embodiment. Further, the memory 360 mayinclude a decoded picture buffer (DPB), and may be constituted by adigital storage medium. The hardware component may further include thememory 360 as an internal/external component.

When a bitstream including video/image information is input, thedecoding apparatus 300 may reconstruct an image correspondingly to aprocess by which video/image information has been processed in theencoding apparatus of FIG. 2 . For example, the decoding apparatus 300may derive units/blocks based on information relating to block partitionobtained from the bitstream. The decoding apparatus 300 may performdecoding by using a processing unit applied in the encoding apparatus.Therefore, the processing unit of decoding may be, for example, a codingunit, which may be partitioned along the quad-tree structure, thebinary-tree structure, and/or the ternary-tree structure from a codingtree unit or a largest coding unit. One or more transform units may bederived from the coding unit. And, the reconstructed image signaldecoded and output through the decoding apparatus 300 may be reproducedthrough a reproducer.

The decoding apparatus 300 may receive a signal output from the encodingapparatus of FIG. 2 in the form of a bitstream, and the received signalmay be decoded through the entropy decoder 310. For example, the entropydecoder 310 may parse the bitstream to derive information (e.g.,video/image information) required for image reconstruction (or picturereconstruction). The video/image information may further includeinformation on various parameter sets such as an adaptation parameterset (APS), a picture parameter set (PPS), a sequence parameter set(SPS), a video parameter set (VPS) or the like. Further, the video/imageinformation may further include general constraint information. Thedecoding apparatus may decode a picture further based on information onthe parameter set and/or the general constraint information. In thepresent disclosure, signaled/received information and/or syntaxelements, which will be described later, may be decoded through thedecoding procedure and be obtained from the bitstream. For example, theentropy decoder 310 may decode information in the bitstream based on acoding method such as exponential Golomb encoding, CAVLC, CABAC, or thelike, and may output a value of a syntax element necessary for imagereconstruction and quantized values of a transform coefficient regardinga residual. More specifically, a CABAC entropy decoding method mayreceive a bin corresponding to each syntax element in a bitstream,determine a context model using decoding target syntax elementinformation and decoding information of neighboring and decoding targetblocks, or information of symbol/bin decoded in a previous step, predictbin generation probability according to the determined context model andperform arithmetic decoding of the bin to generate a symbolcorresponding to each syntax element value. Here, the CABAC entropydecoding method may update the context model using information of asymbol/bin decoded for a context model of the next symbol/bin afterdetermination of the context model. Information on prediction amonginformation decoded in the entropy decoder 310 may be provided to thepredictor (inter predictor 332 and intra predictor 331), and residualvalues, that is, quantized transform coefficients, on which entropydecoding has been performed in the entropy decoder 310, and associatedparameter information may be input to the residual processor 320. Theresidual processor 320 may derive a residual signal (residual block,residual samples, residual sample array). Further, information onfiltering among information decoded in the entropy decoder 310 may beprovided to the filter 350. Meanwhile, a receiver (not shown) whichreceives a signal output from the encoding apparatus may furtherconstitute the decoding apparatus 300 as an internal/external element,and the receiver may be a component of the entropy decoder 310.Meanwhile, the decoding apparatus according to the present disclosuremay be called a video/image/picture coding apparatus, and the decodingapparatus may be classified into an information decoder(video/image/picture information decoder) and a sample decoder(video/image/picture sample decoder). The information decoder mayinclude the entropy decoder 310, and the sample decoder may include atleast one of the dequantizer 321, the inverse transformer 322, the adder340, the filter 350, the memory 360, the inter predictor 332, and theintra predictor 331.

The dequantizer 321 may output transform coefficients by dequantizingthe quantized transform coefficients. The dequantizer 321 may rearrangethe quantized transform coefficients in the form of a two-dimensionalblock. In this case, the rearrangement may perform rearrangement basedon an order of coefficient scanning which has been performed in theencoding apparatus. The dequantizer 321 may perform dequantization onthe quantized transform coefficients using quantization parameter (e.g.,quantization step size information), and obtain transform coefficients.

The deqauntizer 322 obtains a residual signal (residual block, residualsample array) by inverse transforming transform coefficients.

The predictor may perform prediction on the current block, and generatea predicted block including prediction samples for the current block.The predictor may determine whether intra prediction or inter predictionis applied to the current block based on the information on predictionoutput from the entropy decoder 310, and specifically may determine anintra/inter prediction mode.

The predictor may generate a prediction signal based on variousprediction methods. For example, the predictor may apply intraprediction or inter prediction for prediction on one block, and, aswell, may apply intra prediction and inter prediction at the same time.This may be called combined inter and intra prediction (CIIP). Inaddition, the predictor may perform intra block copy (IBC) forprediction on a block. The intra block copy may be used for contentimage/video coding of a game or the like, such as screen content coding(SCC). Although the IBC basically performs prediction in a currentblock, it can be performed similarly to inter prediction in that itderives a reference block in a current block. That is, the IBC may useat least one of inter prediction techniques described in the presentdisclosure.

The intra predictor 331 may predict the current block by referring tothe samples in the current picture. The referred samples may be locatedin the neighbor of or apart from the current block according to theprediction mode. In the intra prediction, prediction modes may include aplurality of non-directional modes and a plurality of directional modes.The intra predictor 331 may determine the prediction mode applied to thecurrent block by using the prediction mode applied to the neighboringblock.

The inter predictor 332 may derive a predicted block for the currentblock based on a reference block (reference sample array) specified by amotion vector on a reference picture. At this time, in order to reducethe amount of motion information transmitted in the inter predictionmode, the motion information may be predicted on a block, subblock, orsample basis based on correlation of motion information between theneighboring block and the current block. The motion information mayinclude a motion vector and a reference picture index. The motioninformation may further include inter prediction direction (L0prediction, L1 prediction, Bi prediction, etc.) information. In the caseof inter prediction, the neighboring block may include a spatialneighboring block existing in the current picture and a temporalneighboring block existing in the reference picture. For example, theinter predictor 332 may configure a motion information candidate listbased on neighboring blocks, and derive a motion vector and/or areference picture index of the current block based on received candidateselection information. Inter prediction may be performed based onvarious prediction modes, and the information on prediction may includeinformation indicating a mode of inter prediction for the current block.

The adder 340 may generate a reconstructed signal (reconstructedpicture, reconstructed block, reconstructed sample array) by adding theobtained residual signal to the prediction signal (predicted block,prediction sample array) output from the predictor 330. When there is noresidual for a processing target block as in a case where the skip modeis applied, the predicted block may be used as a reconstructed block.

The adder 340 may be called a reconstructor or a reconstructed blockgenerator. The generated reconstructed signal may be used for intraprediction of a next processing target block in the current block, andas described later, may be output through filtering or be used for interprediction of a next picture.

Meanwhile, in the picture decoding process, luma mapping with chromascaling (LMCS) may be applied.

The filter 350 may improve subjective/objective video quality byapplying the filtering to the reconstructed signal. For example, thefilter 350 may generate a modified reconstructed picture by applyingvarious filtering methods to the reconstructed picture, and may transmitthe modified reconstructed picture in the memory 360, specifically inthe DPB of the memory 360. The various filtering methods may include,for example, deblocking filtering, sample adaptive offset, an adaptiveloop filter, a bilateral filter or the like.

The (modified) reconstructed picture which has been stored in the DPB ofthe memory 360 may be used as a reference picture in the inter predictor332. The memory 360 may store motion information of a block in thecurrent picture, from which motion information has been derived (ordecoded) and/or motion information of blocks in an already reconstructedpicture. The stored motion information may be transmitted to the interpredictor 260 to be utilized as motion information of a neighboringblock or motion information of a temporal neighboring block. The memory360 may store reconstructed samples of reconstructed blocks in thecurrent picture, and transmit them to the intra predictor 331.

In this specification, the examples described in the predictor 330, thedequantizer 321, the inverse transformer 322, and the filter 350 of thedecoding apparatus 300 may be similarly or correspondingly applied tothe predictor 220, the dequantizer 234, the inverse transformer 235, andthe filter 260 of the encoding apparatus 200, respectively.

As described above, prediction is performed in order to increasecompression efficiency in performing image coding. Through this, apredicted block including prediction samples for a current block, whichis a coding target block, may be generated. Here, the predicted blockincludes prediction samples in a space domain (or pixel domain). Thepredicted block may be identically derived in the encoding apparatus andthe decoding apparatus, and the encoding apparatus may increase imagecoding efficiency by signaling to the decoding apparatus not originalsample value of an original block itself but information on residual(residual information) between the original block and the predictedblock. The decoding apparatus may derive a residual block includingresidual samples based on the residual information, generate areconstructed block including reconstructed samples by adding theresidual block to the predicted block, and generate a reconstructedpicture including reconstructed blocks.

The residual information may be generated through transform andquantization procedures. For example, the encoding apparatus may derivea residual block between the original block and the predicted block,derive transform coefficients by performing a transform procedure onresidual samples (residual sample array) included in the residual block,and derive quantized transform coefficients by performing a quantizationprocedure on the transform coefficients, so that it may signalassociated residual information to the decoding apparatus (through abitstream). Here, the residual information may include valueinformation, position information, a transform technique, transformkernel, a quantization parameter or the like of the quantized transformcoefficients. The decoding apparatus may perform aquantization/dequantization procedure and derive the residual samples(or residual sample block), based on residual information. The decodingapparatus may generate a reconstructed block based on a predicted blockand the residual block. The encoding apparatus may derive a residualblock by dequantizing/inverse transforming quantized transformcoefficients for reference for inter prediction of a next picture, andmay generate a reconstructed picture based on this.

FIG. 4 schematically illustrates a multiple transform techniqueaccording to an embodiment of the present disclosure.

Referring to FIG. 4 , a transformer may correspond to the transformer inthe encoding apparatus of foregoing FIG. 2 , and an inverse transformermay correspond to the inverse transformer in the encoding apparatus offoregoing FIG. 2 , or to the inverse transformer in the decodingapparatus of FIG. 3 .

The transformer may derive (primary) transform coefficients byperforming a primary transform based on residual samples (residualsample array) in a residual block (S410). This primary transform may bereferred to as a core transform. Herein, the primary transform may bebased on multiple transform selection (MTS), and when a multipletransform is applied as the primary transform, it may be referred to asa multiple core transform.

The multiple core transform may represent a method of transformingadditionally using discrete cosine transform (DCT) type 2 and discretesine transform (DST) type 7, DCT type 8, and/or DST type 1. That is, themultiple core transform may represent a transform method of transforminga residual signal (or residual block) of a space domain into transformcoefficients (or primary transform coefficients) of a frequency domainbased on a plurality of transform kernels selected from among the DCTtype 2, the DST type 7, the DCT type 8 and the DST type 1. Herein, theprimary transform coefficients may be called temporary transformcoefficients from the viewpoint of the transformer.

In other words, when the conventional transform method is applied,transform coefficients might be generated by applying transform from aspace domain to a frequency domain for a residual signal (or residualblock) based on the DCT type 2. Unlike to this, when the multiple coretransform is applied, transform coefficients (or primary transformcoefficients) may be generated by applying transform from a space domainto a frequency domain for a residual signal (or residual block) based onthe DCT type 2, the DST type 7, the DCT type 8, and/or DST type 1.Herein, the DCT type 2, the DST type 7, the DCT type 8, and the DST type1 may be called a transform type, transform kernel or transform core.These DCT/DST transform types can be defined based on basis functions.

If the multiple core transform is performed, then a vertical transformkernel and a horizontal transform kernel for a target block may beselected from among the transform kernels, a vertical transform for thetarget block may be performed based on the vertical transform kernel,and a horizontal transform for the target block may be performed basedon the horizontal transform kernel. Here, the horizontal transform mayrepresent a transform for horizontal components of the target block, andthe vertical transform may represent a transform for vertical componentsof the target block. The vertical transform kernel/horizontal transformkernel may be adaptively determined based on a prediction mode and/or atransform index of a target block (CU or sub-block) including a residualblock.

Further, according to an example, if the primary transform is performedby applying the MTS, a mapping relationship for transform kernels may beset by setting specific basis functions to predetermined values andcombining basis functions to be applied in the vertical transform or thehorizontal transform. For example, when the horizontal transform kernelis expressed as trTypeHor and the vertical direction transform kernel isexpressed as trTypeVer, a trTypeHor or trTypeVer value of 0 may be setto DCT2, a trTypeHor or trTypeVer value of 1 may be set to DST7, and atrTypeHor or trTypeVer value of 2 may be set to DCT8.

In this case, MTS index information may be encoded and signaled to thedecoding apparatus to indicate any one of a plurality of transformkernel sets. For example, an MTS index of 0 may indicate that bothtrTypeHor and trTypeVer values are 0, an MTS index of 1 may indicatethat both trTypeHor and trTypeVer values are 1, an MTS index of 2 mayindicate that the trTypeHor value is 2 and the trTypeVer value. Is 1, anMTS index of 3 may indicate that the trTypeHor value is 1 and thetrTypeVer value is 2, and an MTS index of 4 may indicate that both bothtrTypeHor and trTypeVer values are 2.

In one example, transform kernel sets according to MTS index informationare illustrated in the following table.

TABLE 1 tu_mts_idx[x0][y0] 0 1 2 3 4 trTypeHor 0 1 2 1 2 trTypeVer 0 1 12 2

The transformer may derive modified (secondary) transform coefficientsby performing the secondary transform based on the (primary) transformcoefficients (S420). The primary transform is a transform from a spatialdomain to a frequency domain, and the secondary transform refers totransforming into a more compressive expression by using a correlationexisting between (primary) transform coefficients. The secondarytransform may include a non-separable transform. In this case, thesecondary transform may be called a non-separable secondary transform(NSST), or a mode-dependent non-separable secondary transform (MDNSST).The non-separable secondary transform may represent a transform whichgenerates modified transform coefficients (or secondary transformcoefficients) for a residual signal by secondary-transforming, based ona non-separable transform matrix, (primary) transform coefficientsderived through the primary transform. At this time, the verticaltransform and the horizontal transform may not be applied separately (orhorizontal and vertical transforms may not be applied independently) tothe (primary) transform coefficients, but the transforms may be appliedat once based on the non-separable transform matrix. In other words, thenon-separable secondary transform may represent a transform method inwhich the vertical and horizontal components of the (primary) transformcoefficients are not separated, and for example, two-dimensional signals(transform coefficients) are re-arranged to a one-dimensional signalthrough a certain determined direction (e.g., row-first direction orcolumn-first direction), and then modified transform coefficients (orsecondary transform coefficients) are generated based on thenon-separable transform matrix. For example, according to a row-firstorder, M×N blocks are disposed in a line in an order of a first row, asecond row, . . . , and an Nth row. According to a column-first order,M×N blocks are disposed in a line in an order of a first column, asecond column, . . . , and an Nth column. The non-separable secondarytransform may be applied to a top-left region of a block configured with(primary) transform coefficients (hereinafter, may be referred to as atransform coefficient block). For example, if the width (W) and theheight (H) of the transform coefficient block are all equal to orgreater than 8, an 8×8 non-separable secondary transform may be appliedto a top-left 8×8 region of the transform coefficient block. Further, ifthe width (W) and the height (H) of the transform coefficient block areall equal to or greater than 4, and the width (W) or the height (H) ofthe transform coefficient block is less than 8, then a 4×4 non-separablesecondary transform may be applied to a top-left min(8.W)×min(8,H)region of the transform coefficient block. However, the embodiment isnot limited to this, and for example, even if only the condition thatthe width (W) or height (H) of the transform coefficient block is equalto or greater than 4 is satisfied, the 4×4 non-separable secondarytransform may be applied to the top-left min(8.W)×min(8,H) region of thetransform coefficient block.

Specifically, for example, if a 4×4 input block is used, thenon-separable secondary transform may be performed as follows.

The 4×4 input block X may be represented as follows.

$\begin{matrix}{X = \begin{bmatrix}X_{00} & X_{01} & X_{02} & X_{03} \\X_{10} & X_{11} & X_{12} & X_{13} \\X_{20} & X_{21} & X_{22} & X_{23} \\X_{30} & X_{31} & X_{32} & X_{33}\end{bmatrix}} & \left\lbrack {{Equation}1} \right\rbrack\end{matrix}$

If the X is represented in the form of a vector, the vector

may be represented as below.

=[X ₀₀ X ₀₁ X ₀₂ X ₀₃ X ₁₀ X ₁₁ X ₁₂ X ₁₃ X ₂₀ X ₂₁ X ₂₂ X ₂₃ X ₃₀ X ₃₁X ₃₂ X ₃₃]^(T)  [Equation 2]

In Equation 2, the vector

is a one-dimensional vector obtained by rearranging the two-dimensionalblock X of Equation 1 according to the row-first order.

In this case, the secondary non-separable transform may be calculated asbelow.

=T·

  [Equation 3]

In this equation,

represents a transform coefficient vector, and T represents a 16×16(non-separable) transform matrix.

Through foregoing Equation 3, a 16×1 transform coefficient vector

may be derived, and the

may be re-organized into a 4×4 block through a scan order (horizontal,vertical, diagonal and the like). However, the above-describedcalculation is an example, and hypercube-Givens transform (HyGT) or thelike may be used for the calculation of the non-separable secondarytransform in order to reduce the computational complexity of thenon-separable secondary transform.

Meanwhile, in the non-separable secondary transform, a transform kernel(or transform core, transform type) may be selected to be modedependent. In this case, the mode may include the intra prediction modeand/or the inter prediction mode.

As described above, the non-separable secondary transform may beperformed based on an 8×8 transform or a 4×4 transform determined basedon the width (W) and the height (H) of the transform coefficient block.The 8×8 transform refers to a transform that is applicable to an 8×8region included in the transform coefficient block when both W and H areequal to or greater than 8, and the 8×8 region may be a top-left 8×8region in the transform coefficient block. Similarly, the 4×4 transformrefers to a transform that is applicable to a 4×4 region included in thetransform coefficient block when both W and H are equal to or greaterthan 4, and the 4×4 region may be a top-left 4×4 region in the transformcoefficient block. For example, an 8×8 transform kernel matrix may be a64×64/16×64 matrix, and a 4×4 transform kernel matrix may be a16×16/8×16 matrix.

Here, to select a mode-dependent transform kernel, two non-separablesecondary transform kernels per transform set for a non-separablesecondary transform may be configured for both the 8×8 transform and the4×4 transform, and there may be four transform sets. That is, fourtransform sets may be configured for the 8×8 transform, and fourtransform sets may be configured for the 4×4 transform. In this case,each of the four transform sets for the 8×8 transform may include two8×8 transform kernels, and each of the four transform sets for the 4×4transform may include two 4×4 transform kernels.

The sizes of the transforms, the numbers of sets, and the numbers oftransform kernels in each set mentioned above are merely forillustration. Instead, a size other than 8×8 or 4×4 may be used, n setsmay be configured, and k transform kernels may be included in each set.

The transform set may be called an NSST set, and the transform kernel inthe NSST set may be called an NSST kernel. The selection of a specificset from among the transform sets may be performed, for example, basedon the intra prediction mode of the target block (CU or sub-block).

For reference, as an example, the intra prediction mode may include twonon-directional (or non-angular) intra prediction modes and 65directional (or angular) intra prediction modes. The non-directionalintra prediction modes may include a No. 0 planar intra prediction mode,and a No. 1 DC intra prediction mode, and the directional intraprediction modes may include 65 intra prediction modes between a No. 2intra prediction mode and a No. 66 intra prediction mode. However, thisis an example, and the present disclosure may be applied to a case wherethere are different number of intra prediction modes. Meanwhile,according to circumstances, a No. 67 intra prediction mode may befurther used, and the No. 67 intra prediction mode may represent alinear model (LM) mode.

FIG. 5 illustrates directional intra modes of 65 prediction directions.

Referring to FIG. 5 , on the basis of the No. 34 intra prediction modehaving a left upward diagonal prediction direction, the intra predictionmode having a horizontal directionality and the intra prediction modehaving vertical directionality may be classified. H and V of FIG. 5 meanhorizontal directionality and vertical directionality, respectively, andnumerals −32 to 32 indicate displacements in 1/32 units on the samplegrid position. This may represent an offset for the mode index value.The Nos. 2 to 33 intra prediction modes have the horizontaldirectionality, and the Nos. 34 to 66 intra prediction modes have thevertical directionality. Meanwhile, strictly speaking, the No. 34 intraprediction mode may be considered as being neither horizontal norvertical, but it may be classified as belonging to the horizontaldirectionality in terms of determining the transform set of thesecondary transform. This is because the input data is transposed to beused for the vertical direction mode symmetrical on the basis of the No.34 intra prediction mode, and the input data alignment method for thehorizontal mode is used for the No. 34 intra prediction mode.Transposing input data means that rows and columns of two-dimensionalblock data M×N are switched into N×M data. The No. 18 intra predictionmode and the No. 50 intra prediction mode may represent a horizontalintra prediction mode and a vertical intra prediction mode,respectively, and the No. 2 intra prediction mode may be called a rightupward diagonal intra prediction mode because it has a left referencepixel and predicts in a right upward direction. In the same manner, theNo. 34 intra prediction mode may be called a right downward diagonalintra prediction mode, and the No. 66 intra prediction mode may becalled a left downward diagonal intra prediction mode.

In one example, four transform sets may be mapped according to an intraprediction mode as in the following table.

TABLE 2 stPredModeIntra stTrSetIdx stPredModeIntra < 0 1  0 <=stPredModeIntra <= 1 0  2 <= stPredModeIntra <= 12 1 13 <=stPredModeIntra <= 23 2 24 <= stPredModeIntra <= 44 3 45 <=stPredModeIntra <= 55 2 56 <= stPredModeIntra 1

As shown in Table 2, one of the four transform sets, that is,stTrSetIdx, may be mapped to one of four values, that is, 0 to 3,according to the intra prediction mode.

When a specific set is determined to be used for the non-separabletransform, one of k transform kernels in the specific set may beselected through the non-separable secondary transform index. Theencoding apparatus may derive a non-separable secondary transform indexindicating a specific transform kernel based on the rate-distortion (RD)check, and may signal the non-separable secondary transform index to thedecoding apparatus. The decoding apparatus may select one from among ktransform kernels in the specific set based on the non-separablesecondary transform index. For example, the NSST index value 0 mayindicate a first non-separable secondary transform kernel, the NSSTindex value 1 may indicate a second non-separable secondary transformkernel, and the NSST index value 2 may indicate a third non-separablesecondary transform kernel. Alternatively, the NSST index value 0 mayindicate that the first non-separable secondary transform is not appliedto a target block, and the NSST index values 1 to 3 may indicate thethree transform kernels.

The transformer may perform the non-separable secondary transform basedon the selected transform kernels, and may obtain modified (secondary)transform coefficients. As described above, the modified transformcoefficients may be derived as transform coefficients quantized throughthe quantizer, and may be encoded and signaled to the decoding apparatusand transferred to the dequantizer/inverse transformer in the encodingapparatus.

Meanwhile, as described above, if the secondary transform is omitted,(primary) transform coefficients, which are an output of the primary(separable) transform, may be derived as transform coefficientsquantized through the quantizer as described above, and may be encodedand signaled to the decoding apparatus and transferred to thedequantizer/inverse transformer in the encoding apparatus.

The inverse transformer may perform a series of procedures in theinverse order to that in which they have been performed in theabove-described transformer. The inverse transformer may receive(dequantized) transformer coefficients, and derive (primary) transformcoefficients by performing a secondary (inverse) transform (S450), andmay obtain a residual block (residual samples) by performing a primary(inverse) transform on the (primary) transform coefficients (S460). Inthis connection, the primary transform coefficients may be calledmodified transform coefficients from the viewpoint of the inversetransformer. As described above, the encoding apparatus and the decodingapparatus may generate the reconstructed block based on the residualblock and the predicted block, and may generate the reconstructedpicture based on the reconstructed block.

The decoding apparatus may further include a secondary inverse transformapplication determinator (or an element to determine whether to apply asecondary inverse transform) and a secondary inverse transformdeterminator (or an element to determine a secondary inverse transform).The secondary inverse transform application determinator may determinewhether to apply a secondary inverse transform. For example, thesecondary inverse transform may be an NSST or an RST, and the secondaryinverse transform application determinator may determine whether toapply the secondary inverse transform based on a secondary transformflag obtained by parsing the bitstream. In another example, thesecondary inverse transform application determinator may determinewhether to apply the secondary inverse transform based on a transformcoefficient of a residual block.

The secondary inverse transform determinator may determine a secondaryinverse transform. In this case, the secondary inverse transformdeterminator may determine the secondary inverse transform applied tothe current block based on an NSST (or RST) transform set specifiedaccording to an intra prediction mode. In an embodiment, a secondarytransform determination method may be determined depending on a primarytransform determination method. Various combinations of primarytransforms and secondary transforms may be determined according to theintra prediction mode. Further, in an example, the secondary inversetransform determinator may determine a region to which a secondaryinverse transform is applied based on the size of the current block.

Meanwhile, as described above, if the secondary (inverse) transform isomitted, (dequantized) transform coefficients may be received, theprimary (separable) inverse transform may be performed, and the residualblock (residual samples) may be obtained. As described above, theencoding apparatus and the decoding apparatus may generate thereconstructed block based on the residual block and the predicted block,and may generate the reconstructed picture based on the reconstructedblock.

Meanwhile, in the present disclosure, a reduced secondary transform(RST) in which the size of a transform matrix (kernel) is reduced may beapplied in the concept of NSST in order to reduce the amount ofcomputation and memory required for the non-separable secondarytransform.

Meanwhile, the transform kernel, the transform matrix, and thecoefficient constituting the transform kernel matrix, that is, thekernel coefficient or the matrix coefficient, described in the presentdisclosure may be expressed in 8 bits. This may be a condition forimplementation in the decoding apparatus and the encoding apparatus, andmay reduce the amount of memory required to store the transform kernelwith a performance degradation that can be reasonably accommodatedcompared to the existing 9 bits or 10 bits. In addition, the expressingof the kernel matrix in 8 bits may allow a small multiplier to be used,and may be more suitable for single instruction multiple data (SIMD)instructions used for optimal software implementation.

In the present specification, the term “RST” may mean a transform whichis performed on residual samples for a target block based on a transformmatrix whose size is reduced according to a reduced factor. In the caseof performing the reduced transform, the amount of computation requiredfor transform may be reduced due to a reduction in the size of thetransform matrix. That is, the RST may be used to address thecomputational complexity issue occurring at the non-separable transformor the transform of a block of a great size.

RST may be referred to as various terms, such as reduced transform,reduced secondary transform, reduction transform, simplified transform,simple transform, and the like, and the name which RST may be referredto as is not limited to the listed examples. Alternatively, since theRST is mainly performed in a low frequency region including a non-zerocoefficient in a transform block, it may be referred to as aLow-Frequency Non-Separable Transform (LFNST). The transform index maybe referred to as an LFNST index.

Meanwhile, when the secondary inverse transform is performed based onRST, the inverse transformer 235 of the encoding apparatus 200 and theinverse transformer 322 of the decoding apparatus 300 may include aninverse reduced secondary transformer which derives modified transformcoefficients based on the inverse RST of the transform coefficients, andan inverse primary transformer which derives residual samples for thetarget block based on the inverse primary transform of the modifiedtransform coefficients. The inverse primary transform refers to theinverse transform of the primary transform applied to the residual. Inthe present disclosure, deriving a transform coefficient based on atransform may refer to deriving a transform coefficient by applying thetransform.

FIG. 6 is a diagram illustrating an RST according to an embodiment ofthe present disclosure.

In the present disclosure, a “target block” may refer to a current blockto be coded, a residual block, or a transform block.

In the RST according to an example, an N-dimensional vector may bemapped to an R-dimensional vector located in another space, so that thereduced transform matrix may be determined, where R is less than N. Nmay mean the square of the length of a side of a block to which thetransform is applied, or the total number of transform coefficientscorresponding to a block to which the transform is applied, and thereduced factor may mean an R/N value. The reduced factor may be referredto as a reduced factor, reduction factor, simplified factor, simplefactor or other various terms. Meanwhile, R may be referred to as areduced coefficient, but according to circumstances, the reduced factormay mean R. Further, according to circumstances, the reduced factor maymean the N/R value.

In an example, the reduced factor or the reduced coefficient may besignaled through a bitstream, but the example is not limited to this.For example, a predefined value for the reduced factor or the reducedcoefficient may be stored in each of the encoding apparatus 200 and thedecoding apparatus 300, and in this case, the reduced factor or thereduced coefficient may not be signaled separately.

The size of the reduced transform matrix according to an example may beR×N less than N×N, the size of a conventional transform matrix, and maybe defined as in Equation 4 below.

$\begin{matrix}{T_{R \times N} = \left\lbrack {\begin{matrix}t_{11} & t_{12} \\t_{21} & t_{22} \\ & \vdots \\t_{R1} & t_{R2}\end{matrix}\begin{matrix}t_{13} \\t_{23} \\ \\t_{R3}\end{matrix}\begin{matrix} \\\cdots \\ \ddots \\\cdots\end{matrix}\begin{matrix}t_{1N} \\t_{2N} \\ \vdots \\t_{RN}\end{matrix}} \right\rbrack} & \left\lbrack {{Equation}4} \right\rbrack\end{matrix}$

The matrix T in the Reduced Transform block shown in FIG. 6A may meanthe matrix TR×N of Equation 4. As shown in FIG. 6A, when the reducedtransform matrix TR×N is multiplied to residual samples for the targetblock, transform coefficients for the target block may be derived.

In an example, if the size of the block to which the transform isapplied is 8×8 and R=16 (i.e., R/N=16/64=1/4), then the RST according toFIG. 6A may be expressed as a matrix operation as shown in Equation 5below. In this case, memory and multiplication calculation can bereduced to approximately ¼ by the reduced factor.

In the present disclosure, a matrix operation may be understood as anoperation of multiplying a column vector by a matrix, disposed on theleft of the column vector, to obtain a column vector.

$\begin{matrix}{\left\lbrack {\begin{matrix}t_{1,1} & t_{1,2} \\t_{2,1} & t_{2,2} \\ & \vdots \\t_{16,1} & t_{16,2}\end{matrix}\begin{matrix}t_{1,3} \\t_{2,3} \\ \\t_{16,3}\end{matrix}\begin{matrix}\ldots \\ \\ \ddots \\\cdots\end{matrix}\begin{matrix}t_{1,64} \\t_{2,64} \\ \vdots \\t_{16,64}\end{matrix}} \right\rbrack \times \begin{bmatrix}r_{1} \\r_{2} \\ \vdots \\r_{64}\end{bmatrix}} & \left\lbrack {{Equation}5} \right\rbrack\end{matrix}$

In Equation 5, r1 to r64 may represent residual samples for the targetblock and may be specifically transform coefficients generated byapplying a primary transform. As a result of the calculation of Equation5, transform coefficients ci for the target block may be derived, and aprocess of deriving ci may be as in Equation 6.

for i from to R:

ci=0

-   -   for j from 1 to N

ci+=ti,j*rj  [Equation 6]

As a result of the calculation of Equation 6, transform coefficients c1to cR for the target block may be derived. That is, when R=16, transformcoefficients c1 to c16 for the target block may be derived. If, insteadof RST, a regular transform is applied and a transform matrix of 64×64(N×N) size is multiplied to residual samples of 64×1 (N×1) size, thenonly 16 (R) transform coefficients are derived for the target blockbecause RST was applied, although 64 (N) transform coefficients arederived for the target block. Since the total number of transformcoefficients for the target block is reduced from N to R, the amount ofdata transmitted by the encoding apparatus 200 to the decoding apparatus300 decreases, so efficiency of transmission between the encodingapparatus 200 and the decoding apparatus 300 can be improved.

When considered from the viewpoint of the size of the transform matrix,the size of the regular transform matrix is 64×64 (N×N), but the size ofthe reduced transform matrix is reduced to 16×64 (R×N), so memory usagein a case of performing the RST can be reduced by an R/N ratio whencompared with a case of performing the regular transform. In addition,when compared to the number of multiplication calculations N×N in a caseof using the regular transform matrix, the use of the reduced transformmatrix can reduce the number of multiplication calculations by the R/Nratio (R×N).

In an example, the transformer 232 of the encoding apparatus 200 mayderive transform coefficients for the target block by performing theprimary transform and the RST-based secondary transform on residualsamples for the target block. These transform coefficients may betransferred to the inverse transformer of the decoding apparatus 300,and the inverse transformer 322 of the decoding apparatus 300 may derivethe modified transform coefficients based on the inverse reducedsecondary transform (RST) for the transform coefficients, and may deriveresidual samples for the target block based on the inverse primarytransform for the modified transform coefficients.

The size of the inverse RST matrix TN×R according to an example is N×Rless than the size N×N of the regular inverse transform matrix, and isin a transpose relationship with the reduced transform matrix TR×N shownin Equation 4.

The matrix Tt in the Reduced Inv. Transform block shown in FIG. 6B maymean the inverse RST matrix TR×NT (the superscript T means transpose).When the inverse RST matrix TR×NT is multiplied to the transformcoefficients for the target block as shown in FIG. 6B, the modifiedtransform coefficients for the target block or the residual samples forthe target block may be derived. The inverse RST matrix TR×NT may beexpressed as (TR×NT)N×R.

More specifically, when the inverse RST is applied as the secondaryinverse transform, the modified transform coefficients for the targetblock may be derived when the inverse RST matrix TR×NT is multiplied tothe transform coefficients for the target block. Meanwhile, the inverseRST may be applied as the inverse primary transform, and in this case,the residual samples for the target block may be derived when theinverse RST matrix TR×NT is multiplied to the transform coefficients forthe target block.

In an example, if the size of the block to which the inverse transformis applied is 8×8 and R=16 (i.e., R/N=16/64=1/4), then the RST accordingto FIG. 6B may be expressed as a matrix operation as shown in Equation 7below.

$\begin{matrix}{\left\lbrack {\begin{matrix}t_{1,1} & t_{2,1} \\t_{1,2} & t_{2,2} \\t_{1,3} & t_{2,3} \\ \vdots & \vdots \\ & {\vdots} \\t_{1,64} & t_{2,64}\end{matrix}\begin{matrix} \\{\ldots} \\ \\ \\ \ddots \\\cdots\end{matrix}\begin{matrix}t_{16,1} \\t_{16,2} \\t_{16,3} \\ \vdots \\ \vdots \\t_{16,64}\end{matrix}} \right\rbrack \times \begin{bmatrix}c_{1} \\c_{2} \\ \vdots \\c_{16}\end{bmatrix}} & \left\lbrack {{Equation}7} \right\rbrack\end{matrix}$

In Equation 7, c1 to c16 may represent the transform coefficients forthe target block. As a result of the calculation of Equation 7, rjrepresenting the modified transform coefficients for the target block orthe residual samples for the target block may be derived, and theprocess of deriving rj may be as in Equation 8.

-   -   for i from to N:

ri=0

-   -   for j from 1 to R

ri+=tj,i*cj  [Equation 8]

As a result of the calculation of Equation 8, r1 to rN representing themodified transform coefficients for the target block or the residualsamples for the target block may be derived. When considered from theviewpoint of the size of the inverse transform matrix, the size of theregular inverse transform matrix is 64×64 (N×N), but the size of thereduced inverse transform matrix is reduced to 64×16 (R×N), so memoryusage in a case of performing the inverse RST can be reduced by an R/Nratio when compared with a case of performing the regular inversetransform. In addition, when compared to the number of multiplicationcalculations N×N in a case of using the regular inverse transformmatrix, the use of the reduced inverse transform matrix can reduce thenumber of multiplication calculations by the R/N ratio (N×R).

Meanwhile, a transform set configuration shown in Table 2 may also beapplied to an 8×8 RST. That is, the 8×8 RST may be applied according toa transform set in Table 2. Since one transform set includes two orthree transforms (kernels) according to an intra prediction mode, it maybe configured to select one of up to four transforms including that in acase where no secondary transform is applied. In a transform where nosecondary transform is applied, it may be considered to apply anidentity matrix. Assuming that indexes 0, 1, 2, and 3 are respectivelyassigned to the four transforms (e.g., index 0 may be allocated to acase where an identity matrix is applied, that is, a case where nosecondary transform is applied), an NSST index as a syntax element maybe signaled for each transform coefficient block, thereby designating atransform to be applied. That is, through the NSST index, it is possibleto designate an 8×8 NSST for a top-left 8×8 block and to designate an8×8 RST in an RST configuration. The 8×8 NSST and the 8×8 RST refer totransforms applicable to an 8×8 region included in the transformcoefficient block when both W and H of the target block to betransformed are equal to or greater than 8, and the 8×8 region may be atop-left 8×8 region in the transform coefficient block. Similarly, a 4×4NSST and a 4×4 RST refer to transforms applicable to a 4×4 regionincluded in the transform coefficient block when both W and H of thetarget block to are equal to or greater than 4, and the 4×4 region maybe a top-left 4×4 region in the transform coefficient block.

According to an embodiment of the present disclosure, for a transform inan encoding process, only 48 pieces of data may be selected and amaximum 16×48 transform kernel matrix may be applied thereto, ratherthan applying a 16×64 transform kernel matrix to 64 pieces of dataforming an 8×8 region. Here, “maximum” means that m has a maximum valueof 16 in an m×48 transform kernel matrix for generating m coefficients.That is, when an RST is performed by applying an m×48 transform kernelmatrix (m≤16) to an 8×8 region, 48 pieces of data are input and mcoefficients are generated. When m is 16, 48 pieces of data are inputand 16 coefficients are generated. That is, assuming that 48 pieces ofdata form a 48×1 vector, a 16×48 matrix and a 48×1 vector aresequentially multiplied, thereby generating a 16×1 vector. Here, the 48pieces of data forming the 8×8 region may be properly arranged, therebyforming the 48×1 vector. Here, when a matrix operation is performed byapplying a maximum 16×48 transform kernel matrix, 16 modified transformcoefficients are generated, and the 16 modified transform coefficientsmay be arranged in a top-left 4×4 region according to a scanning order,and a top-right 4×4 region and a bottom-left 4×4 region may be filledwith zeros.

For an inverse transform in a decoding process, the transposed matrix ofthe foregoing transform kernel matrix may be used. That is, when aninverse RST or LFNST is performed in an inverse transform processperformed by the decoding apparatus, input coefficient data to which theinverse RST is applied is configured in a one-dimensional vectoraccording to a predetermined arrangement order, and a modifiedcoefficient vector obtained by multiplying the one-dimensional vectorand a corresponding inverse RST matrix on the left of theone-dimensional vector may be arranged in a two-dimensional blockaccording to a predetermined arrangement order.

In summary, in the transform process, when an RST or LFNST is applied toan 8×8 region, a matrix operation of 48 transform coefficients intop-left, top-right, and bottom-left regions of the 8×8 region excludingthe bottom-right region among transform coefficients in the 8×8 regionand a 16×48 transform kernel matrix. For the matrix operation, the 48transform coefficients are input in a one-dimensional array. When thematrix operation is performed, 16 modified transform coefficients arederived, and the modified transform coefficients may be arranged in thetop-left region of the 8×8 region.

On the contrary, in the inverse transform process, when an inverse RSTor LFNST is applied to an 8×8 region, 16 transform coefficientscorresponding to a top-left region of the 8×8 region among transformcoefficients in the 8×8 region may be input in a one-dimensional arrayaccording to a scanning order and may be subjected to a matrix operationwith a 48×16 transform kernel matrix. That is, the matrix operation maybe expressed as (48×16 matrix)*(16×1 transform coefficient vector)=(48×1modified transform coefficient vector). Here, an n×1 vector may beinterpreted to have the same meaning as an n×1 matrix and may thus beexpressed as an n×1 column vector. Further, * denotes matrixmultiplication. When the matrix operation is performed, 48 modifiedtransform coefficients may be derived, and the 48 modified transformcoefficients may be arranged in top-left, top-right, and bottom-leftregions of the 8×8 region excluding a bottom-right region.

When a secondary inverse transform is based on an RST, the inversetransformer 235 of the encoding apparatus 200 and the inversetransformer 322 of the decoding apparatus 300 may include an inversereduced secondary transformer to derive modified transform coefficientsbased on an inverse RST on transform coefficients and an inverse primarytransformer to derive residual samples for a target block based on aninverse primary transform on the modified transform coefficients. Theinverse primary transform refers to the inverse transform of a primarytransform applied to a residual. In the present disclosure, deriving atransform coefficient based on a transform may refer to deriving thetransform coefficient by applying the transform.

Meanwhile, according to an embodiment, a block differential pulse codedmodulation (BDPCM) technique may be used. BDPCM may also be called RDPCM(quantized residual block-based Delta Pulse Code Modulation).

When predicting a block by applying the BDPCM, reconstructed samples areused to predict a row or a column of the block line-by-line. In thiscase, the reference pixels used may be unfiltered samples. The BDPCMdirection may indicate whether a vertical direction or a horizontaldirection prediction is used. The prediction error is quantized in thespatial domain, and the pixel is reconstructed by adding the dequantizedprediction error to the prediction samples. As an alternative to thisBDPCM, a quantized residual domain BDPCM may be proposed, and theprediction direction or signaling may be the same as the BDPCM appliedto the spatial domain. That is, the quantization coefficient itself canbe accumulated like DPCM (Delta Pulse Code Modulation) through thequantized residual domain BDPCM, and then the residual can bereconstructed through dequantization. Therefore, the quantized residualdomain BDPCM may be used in the sense of applying DPCM in the residualcoding. A quantized residual domain used below is quantized residualswithout transformation in which residuals derived based on prediction isquantized, and refers to a domain for a quantized residual samples.

For a block of size M(row)×N(column), let's assume that predictionresiduals obtained by performing intra prediction in the horizontaldirection (copying the left neighboring pixel line to the predictionblock line by line) or the intra prediction in the vertical direction(copying the upper neighboring line to the prediction blockline-by-line) using unfiltered samples among the left or upper boundarysamples are r_((i,j)) (0≤i≤M−1, 0≤j≤N−1). And, suppose that thequantized version of the residual r_((i,j)) is Q(r_((i,j)))(0≤i≤M−1,0≤j≤N−1). Here, the residual means a difference value between the valueof the original block and the value of the prediction block.

Then, when the BDPCM is applied to the quantized residual samples, amodified array {tilde over (R)} of M×N configured with {tilde over(r)}_(i,j) is derived.

When vertical BDPCM is signaled, {tilde over (r)}_(i,j) is as follows.

$\begin{matrix}{{\overset{\_}{r}}_{i,j} = \left\{ {\begin{matrix}{{Q\left( r_{i,j} \right)},} \\{{{Q\left( r_{i,j} \right)} - {Q\left( r_{{({i - 1})},j} \right)}},}\end{matrix}\begin{matrix}{{i = 0},{0 \leq j \leq \left( {N - 1} \right)}} \\{{1 \leq i \leq \left( {M - 1} \right)},{0 \leq j \leq \left( {N - 1} \right)}}\end{matrix}} \right.} & {\left\lbrack {{Equation}9} \right\rbrack}\end{matrix}$

Similarly applied to horizontal prediction, the residual quantizedsamples are as follows.

$\begin{matrix}{{\overset{\sim}{r}}_{i,j} = \left\{ {\begin{matrix}{{Q\left( r_{i,j} \right)},} \\{{{Q\left( r_{i,j} \right)} - {Q\left( r_{i,{({j - 1})}} \right)}},}\end{matrix}\begin{matrix}{{0 \leq i \leq \left( {M - 1} \right)},{j = 0}} \\{{0 \leq i \leq \left( {M - 1} \right)},{1 \leq j \leq \left( {N - 1} \right)}}\end{matrix}} \right.} & \left\lbrack {{Equation}10} \right\rbrack\end{matrix}$

The quantized residual sample ({tilde over (r)}_(i,j)) is transmitted tothe decoding device.

In the decoding apparatus, in order to derive Q(r_((i,j))) (0≤i≤M−1,0≤j≤N−1.), the above operation is performed inversely.

For vertical prediction, the following equation can be applied.

Q(r _(i,j))=Σ_(k=0) ^(i) {tilde over (r)} _(k,j), 0≤i≤(M−1),0≤j≤(N−1).  [Equation 11]

In addition, the following equation may be applied to horizontalprediction.

$\begin{matrix}{{{Q\left( r_{i,j} \right)} = {\sum\limits_{k = 0}^{j}{\overset{\sim}{r}}_{i,k}}},{0 \leq i \leq \left( {M - 1} \right)},{0 \leq j \leq \left( {N - 1} \right)}} & \left\lbrack {{Equation}12} \right\rbrack\end{matrix}$

The dequantized quantized residual Q−1(Q(r_((i,j)))) is summed with theintra block prediction value to derive a reconstructed sample value.

The main advantage of this technique is that the inverse BDPCM can beimmediately performed by simply adding predictors at the time of parsingthe coefficients or after parsing during the parsing of thecoefficients.

As described above, the BDPCM may be applied to a quantized residualdomain, and the quantized residual domain may include quantizedresiduals (or quantized residual coefficients), at this time transformskip may be applied to the residuals. That is, for the residual sample,the transform is skipped and quantization can be applied. Alternatively,the quantized residual domain may include quantized transformcoefficients. A flag for whether the BDPCM is applicable may be signaledat the sequence level (SPS), and this flag may be signaled only when itis signaled that the transform skip mode is enabled in the SPS.

When the BDPCM is applied, intra prediction for the quantized residualdomain is performed on the entire block by sample copy according to aprediction direction similar to the intra prediction direction (eg,vertical prediction or horizontal prediction). The residuals arequantized, and the delta values, i.e., the difference values ({tildeover (r)}_(i,j)) are coded between the quantized residuals and thepredictors in the horizontal or vertical direction (ie, the quantizedresidual in the horizontal or vertical direction).

If the BDPCM is applicable, when the CU size is less than or equal toMaxTsSize (maximum transform skip size) for luma samples, and the CU iscoded with intra prediction, flag information may be transmitted at theCU level. Here, MaxTsSize means the maximum block size for allowing thetransform skip mode. This flag information indicates whether aconventional intra coding or the BDPCM is applied. When the BDPCM isapplied, a BDPCM prediction direction flag indicating whether theprediction direction is a horizontal direction or a vertical directionmay be transmitted. Then, the block is predicted through a conventionalhorizontal or vertical intra prediction process using unfilteredreference samples. The residuals are quantized, and the difference valuebetween each quantized residual and its predictor, for example, thealready quantized residual of the neighboring position in the horizontalor vertical direction according to the BDPCM prediction direction, iscoded.

The syntax elements for the above-described contents and semanticsthereof are shown in a table as follows.

TABLE 3 7.3.2.3 Sequence parameter set RBSP syntax sps_sao_enabled_flagu(1) sps_alf_enabled_flag u(1) if( sps_alf_enabled_flag &&chromaArrayType != 0 )  sps_ccalf_enabled_flag u(1)sps_transform_skip_enabled_flag u(1) if( sps_tranform_skip_enables_flag) {  log2_tranform_skip_max_size_minus2 ue(v)  sps_bdpcm_enabled_flagu(1) 7.4.3.3 Sequence parameter set RBSP semanticssps_bdpcm_enabled_flag equal to 1 specifies that intra_bdpcm_luma_flagand intra_bdpcm_chroma_flag may be present in the coding unit syntax forintra coding units. sps_bdpcm_enabled_flag equal to 0 specifies thatintra_bdpcm_luma_flag and intra_bdpcm_chroma_flag are not present in thecoding unit syntax for intra coding units. When not present, the valueof sps_bdpcm_enabled_flag is inferred to be equal to 0.

Table 3 shows “sps_bdpcm_enabled_flag” signaled in a sequence parameterset (SPS), and when the syntax element “sps_bdpcm_enabled_flag” is 1,flag information indicating whether BDPCM is applied to a coding unit inwhich intra prediction is performed, that is, “intra_bdpcm_luma_flag”and “intra_bdpcm_chroma_flag” Indicates are present in the coding unit.

If the syntax element “sps_bdpcm_enabled_flag” does not exist, its valueis inferred to be 0.

TABLE 4 7.3.10.5Coding unit syntax   if sps_bdpcm_enabled_flag &&    cbWidth <= MaxTsSize && cbHeight <= MaxTsSize )   intra_bdpcm_luma_flag ae(v)   if(intra_bdpcm_luma_flag )   intra_bdpcm_luma_dir_flag ae(v) if ( treeType = = SINGLE_TREE | |treeType = = DUAL_TREE_CHROMA ) &&   ChromaArrayType != 0 ) {  if(pred_mode_plt_flag && treeType = = DUAL_TREE_CHROMA )   palette_coding(x0, y0, cbWidth/SubWidthC, cbHeight/SubHeightC, treeType )  else if(!pred_mode_plt_flag ) {   if ( leu_act_enabled_flag ) (    if(cbWidth/SubWidthC <= MaxTsSize && cbHeight/SubHeightC <= MaxTsSize     && sps_bdpcm_enabled_flag )     intra_bdpcm_chroma_flag ae(v)   if( intra_bdpcm_chroma_flag )     intra_bdpcm_chroma_dir_flag ae(v)7.4.11.5Coding unit semantics intra_bdpcm_luma_flag equal to 1 specifiesthat BDPCM is applied to the current luma coding block at the location(x0, y0 ), i.e. the transform is skipped, the intra luma prediction modeis specified by intra_bdpcm_luma_dir_flag. intra_bdpcm_luma_flag equalto 0 specifies that BDPCM is not applied to the current luma codingblock at the location { x0, y0 }. When intra_bdpcm_luma_flag is notpresent it is inferred to be equal to 0. The variable BdpcmFlag[ x ][ y][ cldx ] is set equal to intra_bdpcm_luma_flag for x = x0.x0 + cbWidth− 1, y = y0..y0 + cbHeight − 1 and cldx = 0. intra_bdpcm_luma_dir_flagequal to 0 specifies that the BDPCM prediction direction is horizontal.intra_bdpcm_luma_dir_flag equal to 1 specifies that the BDPCM predictiondirection is vertical. The variable BdpcmDir[ x ][ y ][ cldx ] is setequal to intra_bdpcm_luma_dir_flag for x = x0..x0 + cbWidth − 1, y =y0..y0 + cbHeight − 1 and cldx = 0. intra_bdpcm_chroma_flag equal to 1specifies that BDPCM is applied to the current chroma coding blocks atthe location { x0, 0 } i.e. the transform is skipped, the intra chromaprediction mode is specified by intra_bdpcm_chroma_dir_flag.intra_bdpcm_chroma_flag equal to 0 specifies that BDPCM is not appliedto the current chroma coding blocks at the location { x0, y0 }. Whenintra_bdpcm_chroma_flag is not present it is inferred to be equal to 0.The variable BdpcmFlag[ x ][ y ][ cldx ] is set equal tointra_bdpcm_chroma_flag for x = x0..x0 + cbWidth − 1, y = y0..y0 +cbHeight − 1 and cldx = 1..2. intra_bdpcm_chroma_dir_flag equal to 0specifies that the BDPCM prediction direction is horizontalintra_bdpcm_chroma_dir_flag equal to 1 specifies that the BDPCMprediction direction is vertical. The variable BdpcmDir[ x ][ y ][ cldx] is set equal to intra_bdpcm_chroma_dir_flag for x = x0..x0 + cbWidth −1, y = y0..y0 + cbHeight − 1 and cldx = 1.2.

As described in Table 3, the syntax elements “intra_bdpcm_luma_flag” and“intra_bdpcm_chroma_flag” of Table 4 indicate whether BDPCM is appliedto the current luma coding block or the current chroma coding block. Ifthe value of “intra_bdpcm_luma_flag” or “intra_bdpcm_chroma_flag” is 1,the transformation for the corresponding coding block is skipped, andthe prediction mode for the coding block mat be set to the horizontal orvertical direction according to “intra_bdpcm_luma_dir_flag” or“intra_bdpcm_chroma_dir_flag” indicating the prediction direction. If“intra_bdpcm_luma_flag” or “intra_bdpcm_chroma_flag” is not present,this value is inferred as 0.

If intra_bdpcm_luma_dir_flag” or “intra_bdpcm_chroma_dir_flag”indicating the prediction direction is 0, it indicates that the BDPCMprediction direction is a horizontal direction, and if the value is 1,it indicates that the BDPCM prediction direction is a verticaldirection.

An intra prediction process based on the flag information is shown in atable as follows.

TABLE 5 8.4.3 Derivation process for luma intra prediction mode Input tothis process are: - a luma location ( xCb, yCb ) specifying the top-leftsample of the current luma coding block  relative to the top-left lumasample of the current picture. - a variable cbWidth specifying the widthof the current coding block in luma samples - a variable cbHeightspecifying the height of the current coding block in luma samples. Inthis process, the luma intra prediction mode IntraPredModeY[ xCb ][ yCb] is derived Table 19 specifies the value for the intra prediction modeIntraPredModeY[ xCb ][ yCb ] and the  associated names TABLE 19Specification of intra prediction mode and associated names Intraprediction mode Associated name 0 INTRA_PLANAR 1 INTRA_DC 2 . . . 66INTRA_ANGULAR2 . . . INTRA_ANGULAR66 81 . . . 83INTRA_LT_CCLM_INTRA_L_CCLM, INTRA_T_CCLM  NOTE-: The intra predictionmoes INTRA_LT_CCLM, INTRA_L_CCLM and INTRA_T_CCLM  are only applicableto chroma components. IntraPredModeY[ xCb ][ yCb ] is derived asfollows: - if intra_luma_not_planar_flag[ xCb ][ yCb ] is equal to 0,IntraPredModeY[ xCb ][ yCb ] is set equal  to INTRA_PLANAR. - Otherwise,if BdpcmFlag[ xCb ][ yCb ][ 0 ] is equal to 1, IntraPredModeY[ xCb ][yCb ] is set equal to BdpcmDir [ xCb ][ yCb ][ 0 ] ? INTRA_ANGULAR50 :INTRA_ANGULAR18.

Table 5 shows the process of deriving the intra prediction mode, and theintra prediction mode (IntraPredModeY[xCb][yCb]) is set to INTRA_PLANARaccording to “Table 19” when intra_luma_not_planar_flag[xCb][yCb] is 0,and when intra_luma_not_planar_flag If xCb][yCb] is 1, it may be set toa vertical mode (INTRA_ANGULAR50) or a horizontal mode (INTRA_ANGULAR18)according to the variable BdpcmDir[xCb][yCb][0].

The variable BdpcmDir[xCb][yCb][0] is set equal to the value ofintra_bdpcm_luma_dir_flag or intra_bdpcm_chroma_dir_flag, as shown inTable 4. Accordingly, the intra prediction mode may be set to ahorizontal mode if the variable BdpcmDir[xCb][yCb][0] is 0, and to avertical mode if it is 1.

In addition, when the BDPCM is applied, a dequantization process can berepresented as shown in Table 6.

TABLE 6 8.4.3 Scaling process for transform coefficients Inputs to thisprocess are:  a luma location { xTby, yTby } specifying the top-leftsample of the current luma transform block  relative to the top-leftluma sample of the current picture  a variable nTbw specifying thetransform block width.  a variable nTbH specifying the transform blockheight.  a variable predMode specifying the prediction mode of thecoding unit  a variable cldx specifying the colour component of thecurrent block. Output of this process is the (nTbW)x(nTbH) array d ofscaled transform coefficients with elements  d[ x ][ y ]. For thederivation of the scaled transform coefficients d[ x ][ y ] with x =0..nTbW − 1, y = 0..nTbH − 1, the following applies ...  When BdpcmFlag[xTbY ][ yYbY ][ cldx ] is equal to 1, dz[ x ][ y ] is modified asfollows:   If BdpcmDir[ xTbY ][ yYbY ][ cldx ] is equal to 0 and x isgreater than 0, the following applies:    dz[ x ][ y ] = Clip3(CoeffMin, CoeffMax, dz[ x − 1 ][ y ] + dz[ x ][ y ])   Otherwise, ifBdpcmDir[ xTbY ][ yTbY ][ cldx ] is equal to 1 and y is greater than 0,the following   applies:    dz[ x ][ y ] = Clip3( CoeffMin, CoeffMax,dz[ x ][ y − 1 ] + dz[ x ][ y ])  The value dnc[ x ][ y ] is derived asfollows:   dnc[ x ][ y ] = ( dz[ x ][ y ] * ls[ x ][ y ] +bdOffset) >>bdshift  The scaled transform coefficient d[ x ][ y ] is derived asfollows:   d[ x ][ y ] = Clip3( CoeffMin, CoeffMax, dnc[ x ][ y ] )

Table 6 shows the dequantization process for transform coefficients(8.4.3 Scaling process for transform coefficients). If theBdpcmFlag[xTbY][yYbY][cIdx] value is 1, the dequantized residual value(d[x] y]) can be derived based on the intermediate variable dz[x][y]. IfBdpcmDir[xTbY][yYbY][cIdx] is 0, that is, when intra prediction isperformed by horizontal mode, the variable dz[x][y] is“dz[x−1][y]+dz[x][y]”. Also, if BdpcmDir[xTbY][yYbY][cIdx] is 1, thatis, when intra prediction is performed by the vertical mode, thevariable dz[x][y] is “dz[x][y−1]+dz[x][y]”. That is, the residual at thespecific location may be derived based on the sum of the residual at theprevious location in the horizontal or vertical direction and a valuereceived as residual information at the specific location. This isbecause when BDPCM is applied, the difference between the residualsample value at a specific position (x, y) (x increases from left toright as a horizontal coordinate, y increases from top to bottom as avertical coordinate, and the position in the 2D block is expressed as(x, y). Also, the specific position represents the (x, y) position whenthe top-left position of the corresponding transformation block is set(0, 0)) and the residual sample value at the previous position ((x−1, y)or (x, y−1)) in the horizontal or vertical direction is signaled asresidual information.

Meanwhile, according to an example, when the BDPCM is applied, aninverse secondary transform that is a non-separable transform, forexample, the LFNST may not be applied. Therefore, when the BDPCM isapplied, an LFNST index (a transform index) signaling may be omitted. Asdescribed above, it is possible to indicate whether or not to apply theLFNST and which transform kernel matrix for the LFNST to be appliedthrough the LFNST index. For example, if the LFNST index value is 0, itindicates that LFNST is not applied, and if the LFNST index value is 1or 2, one of two transform kernel matrices constituting the LFNSTtransform set selected based on the intra prediction mode may bespecified. More specific embodiments related to the BDPCM and the LFNSTmay be applied as follows.

First Embodiment

The BDPCM can be applied only to either the luma component or the chromacomponent. In the case of separately coding the CTU split tree for theluma component and the CTU split tree for the chroma component (eg, adual tree structure in the VVC standard), assuming that BDPCM is appliedonly to the luma component, the LFNST index may be transmitted only whenthe BDPCM is not applied to the luma component, and the LFNST index maybe transmitted to all blocks to which the LFNST can be applied to thechroma component. Conversely, in the dual tree structure, assuming thatBDPCM is applied only to the chroma component, the LFNST index can betransmitted only when BDPCM is not applied to the chroma component, andthe LFNST indexes may be transmitted for all blocks to which LFNST canbe applied for the luma component.

Second Embodiment

When the luma component and the chroma component are coded with the sameCTU split tree, that is, when they share a split type (eg, a single treestructure in the VVC standard), in a block to which the BDPCM isapplied, the LFNST may not be applied to both the luma component and thechroma component. Alternatively, it may be configured to apply the LFNSTto only one component (eg, luma component or chroma component) for ablock to which the BDPCM is applied. In this case, only the LFNST indexfor the corresponding component may be coded and signaled.

Third Embodiment

When the BDPCM is applied only to a specific type of image or partialimage (eg, intra prediction image, intra slice, etc.), it may beconfigured to apply the BDPCM only to the image or partial image of thecorresponding type. For an image or partial image to which the BDPCM isapplied, the LFNST index may be transmitted for each block to which theBDPCM is not applied, and for a type or partial image to which the BDPCMis not applied, the LFNST index for all blocks to which the LFNST may beapplied can be transmitted. Here, the block may be a coding block or atransform block.

Fourth Embodiment

The BDPCM can be applied only to blocks of a specific size or less. Forexample, it can be configured to apply the BDPCM only when a block has awidth of W or less and a height of H or less. Here, W and H may be setto 32, respectively. If the width of a block is W or less and the heightis H or less, so that the BDPCM can be applied, the LFNST index may betransmitted only when a flag indicating whether the BDPCM is applied iscoded as 0 (when BDPCM is not applied).

On the other hand, when the width of a block is greater than W or theheight is greater than H, since the BDPCM is not applied, signaling of aflag indicating whether or not the BDPCM is applied is unnecessary, andthe LFNST index may be transmitted for all blocks to which the LFNST canbe applied.

Fifth Embodiment

Combinations of the above first to fourth embodiments may be applied.For example, 1) the BDPCM is applied only to the luma component, 2) theBDPCM is applied only to intra slices, 3) the BDPCM is configured to beapplied only when both width and height are 32 or less, and the LFNSTindex may be coded and signaled for blocks to which the BDPCM isapplied.

The following drawings are provided to describe specific examples of thepresent disclosure. Since the specific designations of devices or thedesignations of specific signals/messages/fields illustrated in thedrawings are provided for illustration, technical features of thepresent disclosure are not limited to specific designations used in thefollowing drawings.

FIG. 7 is a flowchart illustrating an operation of an image decodingapparatus according to an embodiment of the present disclosure.

Each operation illustrated in FIG. 7 may be performed by the decodingapparatus 300 illustrated in FIG. 3 . Specifically, S710 may beperformed by the entropy decoder 310 illustrated in FIG. 3 , S720 may beperformed by the dequantizer 321 illustrated in FIGS. 3 , S730 and S740may be performed by the inverse transformer 322 illustrated in FIG. 3 ,and S750 may be performed by the adder 340 illustrated in FIG. 3 .Operations according to S710 to S750 are based on some of the foregoingdetails explained with reference to FIG. 4 to FIG. 6 . Therefore, adescription of specific details overlapping with those explained abovewith reference to FIG. 4 to FIG. 6 will be omitted or will be madebriefly.

The decoding apparatus 300 according to an embodiment may derivequantized transform coefficients for a target block from a bitstream(S710). Specifically, the decoding apparatus 300 may decode informationon the quantized transform coefficients for the target block from thebitstream and may derive the quantized transform coefficients for thetarget block based on the information on the quantized transformcoefficients for the target block. The information on the quantizedtransform coefficients for the target block may be included in asequence parameter set (SPS) or a slice header and may include at leastone of information on whether a reduced transform (RST) is applied,information on a reduced factor, information on a minimum transform sizeto which the RST is applied, information on a maximum transform size towhich the RST is applied, information on a reduced inverse transformsize, and information on a transform index indicating any one oftransform kernel matrices included in a transform set.

The decoding apparatus 300 according to an embodiment may derivetransform coefficients by dequantizing the quantized transformcoefficients for the target block (S720).

The decoding apparatus 300 according to an embodiment may derivemodified transform coefficients based on an inverse non-separabletransform or an inverse reduced secondary transform (RST) of thetransform coefficients (S730).

In an example, the inverse non-separable transform or the inverse RSTmay be performed based on an inverse RST transform matrix, and theinverse RST transform matrix may be a nonsquare matrix in which thenumber of columns is less than the number of rows.

In an embodiment, S730 may include decoding a transform index,determining whether a condition for applying an inverse RST is satisfiedbased on the transform index, selecting a transform kernel matrix, andapplying the inverse RST to the transform coefficients based on theselected transform kernel matrix and/or the reduced factor when thecondition for applying the inverse RST is satisfied. In this case, thesize of an inverse RST matrix may be determined based on the reducedfactor.

The decoding apparatus 300 according to an embodiment may deriveresidual samples for the target block based on an inverse transform ofthe modified transform coefficients (S740).

The decoding apparatus 300 may perform an inverse primary transform onthe modified transform coefficients for the target block, in which casea reduced inverse transform may be applied or a conventional separabletransform may be used as the inverse primary transform.

The decoding apparatus 300 according to an embodiment may generatereconstructed samples based on the residual samples for the target blockand prediction samples for the target block (S750).

Referring to S730, it may be identified that the residual samples forthe target block are derived based on the inverse RST of the transformcoefficients for the target block. From the perspective of the size ofthe inverse transform matrix, since the size of a regular inversetransform matrix is N×N but the size of the inverse RST matrix isreduced to N×R, it is possible to reduce memory usage in a case ofperforming the inverse RST by an R/N ratio compared to that in a case ofperforming a regular transform. Further, using the inverse RST matrixcan reduce the number of multiplications (N×R) by the R/N ratio,compared to the number of multiplications N×N in a case of using theregular inverse transform matrix. In addition, since only R transformcoefficients need to be decoded when the inverse RST is applied, thetotal number of transform coefficients for the target block may bereduced from N to R, compared to that in a case where N transformcoefficients needs to be decoded when a regular inverse transform isapplied, thus increasing decoding efficiency. That is, according toS730, the (inverse) transform efficiency and decoding efficiency of thedecoding apparatus 300 may be increased through the inverse RST.

FIG. 8 is a control flowchart illustrating an image decoding methodaccording to an embodiment of the present document.

The decoding device 300 receives coding information such as BDPCMinformation from the bitstream (S810). In addition, the decodingapparatus 300 may be further received transform skip flag informationindicating whether transform skip is applied to the current block,transform index information for inverse secondary transform, that is,inverse non-separable transform, ie, the LFNST index or MTS indexinformation indicating a transform kernel of the inverse primarytransform.

The BDPCM information may include BDPCM flag information indicatingwhether the BDPCM is applied to the current block and directioninformation on a direction in which the BDPCM is performed.

If the BDPCM is applied to the current block, the BDPCM flag value maybe 1, and if the BDPCM is not applied to the current block, the BDPCMflag value may be 0.

When the BDPCM is applied to the current block, the transform skip flagvalue may be inferred as 1, and when the transform skip flag value is 1,the LFNST index value may be inferred as 0 or not received. That is,when the BDPCM is applied to the current block, the transform may not beapplied to the current block.

Meanwhile, the tree type of the current block may be divided into asingle tree (SINGLE_TREE) or a dual tree (DUAL_TREE) depending onwhether the luma block and the corresponding chroma block have anindividual partition structure. When the chroma block has the samepartition structure as the luma block, it may be represented as a singletree, and when the chroma component block has a partition structuredifferent from that of the luma block, it may be represented as a dualtree. According to an example, the BDPCM may be individually applied toa luma block or a chroma block of the current block. If the BDPCM isapplied to the luma block, the transform index for the luma block maynot be received, and if the BDPCM is applied to the chroma block, thetransform index for the chroma block may not be received.

When the tree structure of the current block is the dual tree, the BDPCMcan be applied to only one component block, and even when the currentblock has the single tree structure, the BDPCM can be applied to onlyone component block. In this case, the LFNST index may be received onlyfor component blocks to which the BDPCM is not applied.

Alternatively, according to an example, the BDPCM may be applied onlywhen the width of the current block is less than or equal to the firstthreshold and the height of the current block is less than or equal tothe second threshold. The first threshold value and the second thresholdvalue may be 32, and may be set to a maximum height or a maximum widthof a transformation block in which transformation is performed.

Meanwhile, the direction information for the BDPCM may indicate ahorizontal direction or a vertical direction, and quantizationinformation may be derived according to the direction information and aprediction sample may be derived according to the direction information.

The decoding apparatus 300 may derive quantized transform coefficientsfor the current block based on the BDPCM (S820). Here, the transformcoefficients may be an untransformed residual sample values.

When the BDPCM is applied to the current block, residual informationreceived by the decoding apparatus 300 may be a difference value of aquantized residual. Depending on the BDPCM direction, a difference valuebetween a quantized residual in a previous vertical line or a previoushorizontal direction line and a quantized residual of a specific linemay be received, and the decoding apparatus 300 may derive the quantizedresidual of the specific line by adding the quantized residual value ofthe previous vertical or horizontal line to the difference value of thereceived quantized residual. The quantized residual may be derived basedon Equation 11 or Equation 12.

The decoding apparatus 300 may derive transform coefficients byperforming the dequantization on the quantized transform coefficients(S830), and may derive residual samples based on the transformcoefficients (S840).

As described above, when the BDPCM is applied to the current block, thedequantized transform coefficient may be derived as a residual samplewithout the transform process.

The intra prediction unit 331 may perform intra prediction on thecurrent block based on the direction in which the BDPCM is performed(S850).

If the BDPCM is applied to the current block, intra prediction using thesame may be performed, which may mean that the BDPCM may be applied onlyto an intra slice or an intra coding block predicted in the intra mode.

The intra prediction is performed based on the direction information forthe BDPCM, and the intra prediction mode of the current block may beeither a horizontal direction mode or a vertical direction mode.

The decoding apparatus 300 may generate a reconstructed picture based onthe derived residual samples and the predicted samples as in S750 ofFIG. 7 (S860).

The following drawings are provided to describe specific examples of thepresent disclosure. Since the specific designations of devices or thedesignations of specific signals/messages/fields illustrated in thedrawings are provided for illustration, technical features of thepresent disclosure are not limited to specific designations used in thefollowing drawings.

FIG. 9 is a flowchart illustrating an operation of a video encodingapparatus according to an embodiment of the present disclosure.

Each operation illustrated in FIG. 9 may be performed by the encodingapparatus 200 illustrated in FIG. 2 . Specifically, S910 may beperformed by the predictor 220 illustrated in FIG. 2 , S820 may beperformed by the subtractor 231 illustrated in FIGS. 2 , S930 and S940may be performed by the transformer 232 illustrated in FIG. 2 , and S950may be performed by the quantizer 233 and the entropy encoder 240illustrated in FIG. 2 . Operations according to S910 to S950 are basedon some of contents described in FIG. 4 to FIG. 6 . Therefore, adescription of specific details overlapping with those explained abovewith reference to FIG. 4 to FIG. 6 will be omitted or will be madebriefly.

The encoding apparatus 200 according to an embodiment may deriveprediction samples based on an intra prediction mode applied to a targetblock (S910).

The encoding apparatus 200 according to an embodiment may deriveresidual samples for the target block (S920).

The encoding apparatus 200 according to an embodiment may derivetransform coefficients for the target block based on primary transformof the residual sample (S930). The primary transform may be performedthrough a plurality of transform kernels, and the transform kernels maybe selected based on the intra prediction mode.

The decoding apparatus 300 may perform a secondary transform or anon-separable transform, specifically an NSST, on the transformcoefficients for the target block, in which case the NSST may beperformed based on a reduced transform (RST) or without being based onthe RST. When the NSST is performed based on the reduced transform, anoperation according to S940 may be performed.

The encoding apparatus 200 according to an embodiment may derivemodified transform coefficients for the target block based on the RST ofthe transform coefficients (S940). In an example, the RST may beperformed based on a reduced transform matrix or a transform kernelmatrix, and the reduced transform matrix may be a non-square matrix inwhich the number of rows is less than the number of columns.

In an embodiment, S940 may include determining whether a condition forapplying the RST is satisfied, generating and encoding the transformindex based on the determination, selecting a transform kernel, andapplying the RST to the residual samples based on the selected transformkernel matrix and/or a reduced factor when the condition for applyingthe RST is satisfied. In this case, the size of the reduced transformkernel matrix may be determined based on the reduced factor.

The encoding apparatus 200 according to an embodiment may derivequantized transform coefficients by performing quantization based on themodified transform coefficients for the target block and may encodeinformation on the quantized transform coefficients (S950).

Specifically, the encoding apparatus 200 may generate the information onthe quantized transform coefficients and may encode the generatedinformation on the quantized transform coefficients.

In an example, the information on the quantized transform coefficientsmay include at least one of information on whether the RST is applied,information on the reduced factor, information on a minimum transformsize to which the RST is applied, and information on a maximum transformsize to which the RST is applied.

Referring to S940, it may be identified that the transform coefficientsfor the target block are derived based on the RST of the residualsamples. From the perspective of the size of the transform kernelmatrix, since the size of a regular transform kernel matrix is N×N butthe size of the reduced transform matrix is reduced to R×N, it ispossible to reduce memory usage in a case of performing the RST by anR/N ratio compared to that in a case of performing a regular transform.Further, using the reduced transform kernel matrix can reduce the numberof multiplications (R×N) by the R/N ratio, compared to the number ofmultiplications N×N in a case of using the regular transform kernelmatrix. In addition, since only R transform coefficients are derivedwhen the RST is applied, the total number of transform coefficients forthe target block may be reduced from N to R, compared to that in a casewhere N transform coefficients are derived when a regular transform isapplied, thus reducing the amount of data transmitted by the encodingapparatus 200 to the decoding apparatus 300. That is, according to S940,the transform efficiency and coding efficiency of the encoding apparatus320 may be increased through the RST.

FIG. 10 is a control flowchart illustrating an image encoding methodaccording to an embodiment of this document.

The encoding apparatus 200 may derive prediction samples for the currentblock based on the BDPCM (S1010).

The encoding apparatus 200 may derive intra prediction samples for thecurrent block based on a specific direction in which the BDPCM isperformed. The specific direction may be a vertical direction or ahorizontal direction, and the prediction samples for the current blockmay be generated according to the intra prediction mode.

Meanwhile, the tree type of the current block may be divided into asingle tree (SINGLE_TREE) or a dual tree (DUAL_TREE) depending onwhether the luma block and the corresponding chroma block have anindividual partition structure. When the chroma block has the samepartition structure as the luma block, it may be represented as a singletree, and when the chroma component block has a partition structuredifferent from that of the luma block, it may be represented as a dualtree. According to an example, the BDPCM may be individually applied toa luma block or a chroma block of the current block.

When the tree structure of the current block is the dual tree, the BDPCMcan be applied to only one component block, and even when the currentblock has the single tree structure, the BDPCM can be applied to onlyone component block.

Alternatively, according to an example, the BDPCM may be applied onlywhen the width of the current block is less than or equal to the firstthreshold and the height of the current block is less than or equal tothe second threshold. The first threshold value and the second thresholdvalue may be 32, and may be set to a maximum height or a maximum widthof a transformation block in which transformation is performed.

The encoding apparatus 200 may derive residual samples for the currentblock based on the prediction samples (S1020) and perform quantizationon the residual samples (S1030).

Then, the encoding apparatus 200 may derive quantized residualinformation based on the BDPCM (S1040).

The encoding apparatus 200 may derive a quantized residual sample of aspecific line, and a difference value between a quantized residualsample of a previous vertical or horizontal line and a quantizedresidual sample of the specific line as quantized residual information.That is, the difference value of a quantized residual, not a normalresidual, is generated as residual information, and may be derived basedon Equation 9 or Equation 10.

The encoding apparatus 200 may encode the quantized residual informationand the coding information for the current block (S1050).

The encoding device 200 may encode BDPCM information, transform skipflag information indicating whether transform skip is applied to thecurrent block, transform index information for inverse secondarytransform, that is, inverse non-separable transform, ie, the LFNST indexor MTS index information indicating a transform kernel of the inverseprimary transform.

The BDPCM information may include BDPCM flag information indicatingwhether the BDPCM is applied to the current block and directioninformation on a direction in which the BDPCM is performed.

When the BDPCM is applied to the current block, the BDPCM flag value maybe encoded as 1, and when the BDPCM is not applied to the current block,the BDPCM flag value may be encoded as 0.

If the BDPCM is applied to the current block, the transform skip flagvalue may be inferred as 1 or encoded as 1. In addition, if thetransform skip flag value is 1, the LFNST index value may be inferred as0 or may not be encoded. That is, when the BDPCM is applied to thecurrent block, the transform may not be applied to the current block.

Also, as described above, when the tree structure of the current blockis a dual tree, the BDPCM can be applied to only one component block,and even when the current block has a single tree structure, the BDPCMcan be applied to only one component block. In this case, the LFNSTindex may be encoded only for component blocks to which the BDPCM is notapplied.

Direction information for the BDPCM may indicate a horizontal directionor a vertical direction.

In the present disclosure, at least one of quantization/dequantizationand/or transform/inverse transform may be omitted. Whenquantization/dequantization is omitted, a quantized transformcoefficient may be referred to as a transform coefficient. Whentransform/inverse transform is omitted, the transform coefficient may bereferred to as a coefficient or a residual coefficient, or may still bereferred to as a transform coefficient for consistency of expression.

In addition, in the present disclosure, a quantized transformcoefficient and a transform coefficient may be referred to as atransform coefficient and a scaled transform coefficient, respectively.In this case, residual information may include information on atransform coefficient(s), and the information on the transformcoefficient(s) may be signaled through a residual coding syntax.Transform coefficients may be derived based on the residual information(or information on the transform coefficient(s)), and scaled transformcoefficients may be derived through inverse transform (scaling) of thetransform coefficients. Residual samples may be derived based on theinverse transform (transform) of the scaled transform coefficients.These details may also be applied/expressed in other parts of thepresent disclosure.

In the above-described embodiments, the methods are explained on thebasis of flowcharts by means of a series of steps or blocks, but thepresent disclosure is not limited to the order of steps, and a certainstep may be performed in order or step different from that describedabove, or concurrently with another step. Further, it may be understoodby a person having ordinary skill in the art that the steps shown in aflowchart are not exclusive, and that another step may be incorporatedor one or more steps of the flowchart may be removed without affectingthe scope of the present disclosure.

The above-described methods according to the present disclosure may beimplemented as a software form, and an encoding apparatus and/ordecoding apparatus according to the disclosure may be included in adevice for image processing, such as, a TV, a computer, a smartphone, aset-top box, a display device or the like.

When embodiments in the present disclosure are embodied by software, theabove-described methods may be embodied as modules (processes, functionsor the like) to perform the above-described functions. The modules maybe stored in a memory and may be executed by a processor. The memory maybe inside or outside the processor and may be connected to the processorin various well-known manners. The processor may include anapplication-specific integrated circuit (ASIC), other chipset, logiccircuit, and/or a data processing device. The memory may include aread-only memory (ROM), a random access memory (RAM), a flash memory, amemory card, a storage medium, and/or other storage device. That is,embodiments described in the present disclosure may be embodied andperformed on a processor, a microprocessor, a controller or a chip. Forexample, function units shown in each drawing may be embodied andperformed on a computer, a processor, a microprocessor, a controller ora chip.

Further, the decoding apparatus and the encoding apparatus to which thepresent disclosure is applied, may be included in a multimediabroadcasting transceiver, a mobile communication terminal, a home cinemavideo device, a digital cinema video device, a surveillance camera, avideo chat device, a real time communication device such as videocommunication, a mobile streaming device, a storage medium, a camcorder,a video on demand (VoD) service providing device, an over the top (OTT)video device, an Internet streaming service providing device, athree-dimensional (3D) video device, a video telephony video device, anda medical video device, and may be used to process a video signal or adata signal. For example, the over the top (OTT) video device mayinclude a game console, a Blu-ray player, an Internet access TV, a Hometheater system, a smartphone, a Tablet PC, a digital video recorder(DVR) and the like.

In addition, the processing method to which the present disclosure isapplied, may be produced in the form of a program executed by acomputer, and be stored in a computer-readable recording medium.Multimedia data having a data structure according to the presentdisclosure may also be stored in a computer-readable recording medium.The computer-readable recording medium includes all kinds of storagedevices and distributed storage devices in which computer-readable dataare stored. The computer-readable recording medium may include, forexample, a Blu-ray Disc (BD), a universal serial bus (USB), a ROM, aPROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppydisk, and an optical data storage device. Further, the computer-readablerecording medium includes media embodied in the form of a carrier wave(for example, transmission over the Internet). In addition, a bitstreamgenerated by the encoding method may be stored in a computer-readablerecording medium or transmitted through a wired or wirelesscommunication network. Additionally, the embodiments of the presentdisclosure may be embodied as a computer program product by programcodes, and the program codes may be executed on a computer by theembodiments of the present disclosure. The program codes may be storedon a computer-readable carrier.

FIG. 11 illustrates the structure of a content streaming system to whichthe present disclosure is applied.

Further, the contents streaming system to which the present disclosureis applied may largely include an encoding server, a streaming server, aweb server, a media storage, a user equipment, and a multimedia inputdevice.

The encoding server functions to compress to digital data the contentsinput from the multimedia input devices, such as the smart phone, thecamera, the camcoder and the like, to generate a bitstream, and totransmit it to the streaming server. As another example, in a case wherethe multimedia input device, such as, the smart phone, the camera, thecamcoder or the like, directly generates a bitstream, the encodingserver may be omitted. The bitstream may be generated by an encodingmethod or a bitstream generation method to which the present disclosureis applied. And the streaming server may store the bitstream temporarilyduring a process to transmit or receive the bitstream.

The streaming server transmits multimedia data to the user equipment onthe basis of a user's request through the web server, which functions asan instrument that informs a user of what service there is. When theuser requests a service which the user wants, the web server transfersthe request to the streaming server, and the streaming server transmitsmultimedia data to the user. In this regard, the contents streamingsystem may include a separate control server, and in this case, thecontrol server functions to control commands/responses betweenrespective equipments in the content streaming system.

The streaming server may receive contents from the media storage and/orthe encoding server. For example, in a case the contents are receivedfrom the encoding server, the contents may be received in real time. Inthis case, the streaming server may store the bitstream for apredetermined period of time to provide the streaming service smoothly.

For example, the user equipment may include a mobile phone, a smartphone, a laptop computer, a digital broadcasting terminal, a personaldigital assistant (PDA), a portable multimedia player (PMP), anavigation, a slate PC, a tablet PC, an ultrabook, a wearable device(e.g., a watch-type terminal (smart watch), a glass-type terminal (smartglass), a head mounted display (HMD)), a digital TV, a desktop computer,a digital signage or the like. Each of servers in the contents streamingsystem may be operated as a distributed server, and in this case, datareceived by each server may be processed in distributed manner.

Claims disclosed herein can be combined in a various way. For example,technical features of method claims of the present disclosure can becombined to be implemented or performed in an apparatus, and technicalfeatures of apparatus claims can be combined to be implemented orperformed in a method. Further, technical features of method claims andapparatus claims can be combined to be implemented or performed in anapparatus, and technical features of method claims and apparatus claimscan be combined to be implemented or performed in a method.

1-15. (canceled)
 16. An image decoding method performed by a decodingapparatus, the method comprising: deriving whether a BDPCM (Block-basedDelta Pulse Code Modulation) is applied to a current block; derivingquantized transform coefficients for the current block based on theBDPCM; deriving transform coefficients by performing a dequantization onthe quantized transform coefficients; and deriving residual samplesbased on the transform coefficients; wherein the deriving whether theBDPCM is applied to the current block is comprising determining a widthof the current block is less than or equal to a first threshold and aheight of the current block is less than or equal to a second threshold,wherein based on the width of the current block being less than or equalto the first threshold and the height of the current block being lessthan or equal to the second threshold, the BDPCM is applied to thecurrent block, wherein based on the BDPCM being applied to the currentblock, an inverse non-separable transform is not applied to thetransform coefficients, wherein based on the BDPCM being applied to thecurrent block, a value of a transform index for the inversenon-separable transform that is applied to the current block is inferredto be
 0. 17. The image decoding method of claim 16, wherein the BDPCM isindividually applied to a luma block of the current block or a chromablock of the current block, wherein based on the BDPCM being applied tothe luma block, a transform index for the luma block is not received,and wherein based on the BDPCM being applied to the chroma block, atransform index for the chroma block is not received.
 18. The imagedecoding method of claim 16, wherein when the BDPCM is applied to thecurrent block, a value of a transform skip flag indicating whether atransform is skipped in the current block is inferred to be
 1. 19. Theimage decoding method of claim 16, wherein the first threshold and thesecond threshold are
 32. 20. The image decoding method of claim 16,wherein quantized transform coefficients are derived based on directioninformation on a direction in which the BDPCM is performed.
 21. Theimage decoding method of claim 20, further comprising: performing anintra prediction on the current block based on the direction in whichthe BDPCM is performed.
 22. The image decoding method of claim 21wherein the direction information indicates a horizontal direction or avertical direction.
 23. An image encoding method performed by an imageencoding apparatus, the method comprising: deriving prediction samplesfor a current block based on a BDPCM (Block-based Delta Pulse CodeModulation); deriving residual samples for the current block based onthe prediction samples; performing quantization on the residual samples;deriving quantized residual information based on the BDPCM; and encodingthe quantized residual information and flag information related to theBDPCM for the current block; wherein based on a width of the currentblock being less than or equal to a first threshold and a height of thecurrent block being less than or equal to a second threshold, the flaginformation is encoded; wherein based on the BDPCM being applied to thecurrent block, a non-separable transform is not applied to the currentblock, and wherein based on the BDPCM being applied to the currentblock, a transform index for the non-separable transform is not encoded.24. The image encoding method of claim 23, wherein the BDPCM isindividually applied to a luma block of the current block or a chromablock of the current block, wherein based on the BDPCM being applied tothe luma block, a transform index for the luma block is not encoded, andwherein based on the BDPCM being applied to the chroma block, atransform index for the chroma block is not encoded.
 25. The imageencoding method of claim 23, wherein the first threshold and the secondthreshold are
 32. 26. The image encoding method of claim 23, whereinintra prediction samples for the current block is derived based on aspecific direction in which the BDPCM is performed, and wherein thequantization on the residual samples is performed based on the specificdirection.
 27. The image encoding method of claim 26, wherein thespecific direction includes a horizontal direction or a verticaldirection.
 28. A non-transitory computer-readable digital storage mediumstoring a bitstream generated by a method, the method comprising:deriving prediction samples for a current block based on a BDPCM(Block-based Delta Pulse Code Modulation); deriving residual samples forthe current block based on the prediction samples; performingquantization on the residual samples; deriving quantized residualinformation based on the BDPCM; and encoding the quantized residualinformation and flag information related to the BDPCM for the currentblock to generate the bitstream; wherein based on a width of the currentblock being less than or equal to a first threshold and a height of thecurrent block being less than or equal to a second threshold, the flaginformation is encoded; wherein based on the BDPCM being applied to thecurrent block, a non-separable transform is not applied to the currentblock, and wherein based on the BDPCM being applied to the currentblock, a transform index for the non-separable transform is not encoded.